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tThis  report  documents  the  theory,  structure,  and  implementation  of  a performance  measurement  processor 
(written  in  FORTRAN  IV)  that  can  accept  performance  demonstration  data  representing  various  levels  cf  operator’s 
drill  and,  under  user'  control^  analyze  data  to  provide  candidate  performance  measures  and  validation  test  results. 
The  processor  accepts  two  types  of  information:  (1)  Sample  performance  data  on  magnetic  tape,  and  (2)  User 
information  reflecting  knowledge  about  features  of  the  performance  that  are  considered  to  be  important  to  s 
measurement.  The  sample  performance  data  input  is  smoothed  by  the  processor  in  order  to  remove  or  reduce  noise  • j 
factors  in  accordance  with  information  provided  by  the  useti  Criterion  performance  functions  are,  optionally,  ( 
provided  by  the  user  or  are  computed  by  the  processor  using  drilled  performers’  data.  The  processor  then  develops  a -4- 
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etc  representation  of  the  continuous  performance  data  based  on  observed  deviations  from  the  criterion 
functions.'^hhdiscrete  representation,  in  turn,  ii  used  to  model  each  performance  using  state-space  techniques.  The 
processor  operates  oh  the  state-space  model  to  compute  vectors  which  form  generators  of  various  conceivable 
measure  spaces', Candidate  performance  measures  are  tMp>  generated  by  operating  on  the  vectors  with  multiple 
regression  algorithms.  Empirical  validation  testa  of  several  types  are  applied  to  the  candidate  measures  for  assessment 
of  their  vaHdity-likeHhood. 

The  processor  can  be  applied  to  measurement  problems  where  the  human  operator  working  with  his 
equipment  obtains  demonstrations  of  various  levels  of  performance.  These  potential  applications  include  those 
situations  where  criterion  performance  cannot  be  quantitatively  predefined  and/or  the  existing  definitions  are 
ambiguous. 

Demonstration  of  some  -portions  of  the  processor  was  accomplished  using  limited  flight  demonstration  data 
from  an  instrumented  T-37B  aircraft  for  five  undergraduate  pilot  training  (UPT)  maneuver#  (1)  Barrel  Roll,  (2)  lazy 
8,  (3)  Cloverieaf,  (4)  SphtS,  and  (5)  Normal  Landing. 
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SUMMARY 


Problem 

The  problem  was  to  develop  and  implement  standardized  techniques  for  deriving  and  validating 
measures  of  operator  performance.  Traditional  techniques  involve  hand-selecting  measures  which  appear  to 
have  content  validity,  then  testing  the  measures  against  other  validation  criteria  using  operator  performance 
data.  This  usually  results  in  a resource-consuming  iterative  research  process  that  is  often  unsuccessful, 
because:  (1)  it  is  never  known  at  the  onset  whether  or  not  the  most  useful  measures  have  been  overlooked, 
(2)  the  number  and  potential  validity  of  measures  investigated  are  limited  by  and  vary  with  the  researcher’s 
ingenuity  and  the  time  he  has  available  for  the  study,  and  (3)  the  research  process  and  all  associated  manual 
effort  must  be  repeated  for  each  new  measurement  task. 

Approach 

The  approach  was  to  deyelop  and  implement  computer-aided  techniques  for  deriving  and  validating 
operator  performance  measures.  A “universal”  set  of  potential  measures  was  defined  which  possesses 
characteristics  encompassing  many  traditionally  selected  measures.  The  set  also  inherently  contains  a 
myriad  of  other  measures  whose  characteristics  render  them  reasonable  candidates.  Vectors  were  then 
identified  which  constitute  generators  for  the  set  of  measures  (i.e.,  the  vectors  span  the  defined  measure 
space).  Computational  algorithms  were  developed  which  generate  and  operate  on  the  constituent  vectors 
using  multiple  regression  techniques.  Several  empirical  validation  methods  were  developed  for  testing 
candidate  measures  thereby  generated.  All  techniques  were  implemented  in  a computer-aided  measurement 
processor  which:  (1)  accepts  sample  performance  data  and  various  user  inputs,  and  (2)  generates  and  tests 
candidate  measures,  computes’  statistics  for  assessing  their  validity  likelihood,  and  prints  results  for  user 
analysis. 

Results 

The  developed  measurement  processor  was  successfully  implemented  on  a Sigma  5 computer. 
Demonstrations  of  the  operation  of  the  software  were  performed  using  a limited  amount  of  pilot 
performance  data  recorded  on  a T-37B  aircraft.  The  processor  performed  necessary  data  smoothing, 
automatically  segmented  the  flight  maneuvers  for  measurement,  and  developed  criterion  functions  from  the 
skilled  operator  data  provided.  Actual  generation  and  validation  of  measures  was  not  demonstrable  due  to 
nonavailability  of  originally  anticipated  data.  However,  correct  software  performance  of  all  parts  of  the 
processor  was  verified. 

Conclusions 

The  theoretical  concepts  and  computational  techniques  underlying  the  developed  measurement 
processor  are  unique  and  have  great  potential  for  operator  performance  measurement  research.  The  applied 
concept  of  developing  a set  of  vectors  which  span  a conceived  measure  space  and  operating  on  it  with 
regression  techniques  to  generate  candidate  measures  is  itself  suggestive  of  a new  and  extremely  powerful 
measurement  tool.  The  processor  operation  can  be  largely  independent  of  user  intervention;  however,  it  is 
also  capable  of  accepting  user  inputs  reflecting  his  knowledge  about  specific  measurement  problems.  It 
represents  a truly  interactive  research  system  wherein  user  tasks  as  distinguished  from  processor  tasks  are 
logically  defined,  and  the  outcomes  of  each  are  integrated. 

Evaluation  of  the  adequacy  of  the  spanned  measure  set,  the  generating  vectors,  and  the  computational 
mechanisms  for  generating  and  testing  measures  could  not  be  performed  as  originally  planned  due  to 
nontechnical  problems  which  prevented  the  collection  of  required  data.  This  was  extremely  detrimental  to 
the  study  because:  (!)  many  of  the  techniques  could  not  even  receive  preliminary  test  prior  to  their 
incorporation, in  the  processor,  and  (2)  the  contributions  made  by  this  study' to  the  gcneral  technology  can 
only  be  suggested  instead  of  exemplified. 

Follow-up  research  should  include  derivation  of  the  basis  of  the  defined  measure  set  using  the 
implemented/ processor  as  an  aid  to  empirical  studies.  This  is,  in  essence,  the  real  crux  of  the  operator 
performance  measurement  problem. 
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COMPUTER-AIDED  TECHNIQUES  FOR  PROVIDING 
OPERATOR  PERFORMANCE  MEASURES 


L INTRODUCTION 

This  report  describes  undeiiying  theoretical  concepts  and  computer-implemented  techniques  for 
deriving  valid  and  objective  operator  performance  measures.  The  original  impetus  for  the  work  came  from 
requirements  for  measures  of  pilot  performance;  however  the  techniques  and  development-concepts  are 
equally  applicable  to  general  assessment  of  operator  performance  on  continuous  control  tasks.  Therefore, 
the  basic  mathematical  and  computer  techniques  will  be  described  in  a general  context,  while  (limited) 
example  data  are  presented  for  selected  pilot  performance  tasks. 

A Tale  to  Illustrate  Basic  Concepts 

Suppose  we  are  faced  with  the  relatively  simple  task  of  deriving  and  validating  measures  of 
performance  on  the  terminal  portions  of  a ground  controlled  approach  (GCA).  Let  us  further  simplify  the 
problem  by  restricting  it  to  measurement  of  the  pilot’s  ability  to  maintain  proper  altitude  during  descent  to 
the  runway.  Following  a typically  employed  course  of  action,  we  might  begin  by  specifying  an  intuitive 
(accepted)  notion  of  “ideal”  performance  and  perhaps  sketch  some  hypothetical  performance  profiles,  such 
as  those  shown  in  Figure  1. 

The  next  step  would  typically  be  to  identify  candidate  measures  which,  singly  or  in  combination,  and 
in  whole  or  in  part,  ws  expect  will  solve  the  problem.  Thus,  we  might  logically  pick:  (1)  RMS  Glideslopc 
deviation,  (2)  Maximum  glideslope  deviation,  and  (3)  Time  in  tolerance  (glideslope  t A)  as  measures  to 
compute  and  examine  for  validity. 

We  would  then  probably  perform  some  initial  data  collection,  and  study  the  behavior  of  these 
selected  measures  for  various  pilots  (perhaps  some  novice  and  some  at  various  other  stages  in  the  range  from 
novice  to  experienced).  We  might  disoover  (again  assunring  a typical  case)  that  one  of  the  selected  measures 
tends  to  discriminate  between  some  of  the  novice  and  highly  experienced  performers,  but  not  in  all  cases; 
and  that  none  of  the  measures  say  anything  conclusive  or  consistent  about  performers  whose  experience 
level  (and/or  subjectively  judged  skill  level)  lies  between  the  two  extremes. 

“Aha!”  says  our  colleague.  “The  reason  your  RMS  doesn’t  work  well  is  because  glideslope  deviations 
close  to  the  ground  are  more  critical  than  deviations  at  higher  altitudes.  You  need  to  take  altitude  into 
consideration  and  weight  the  deviations  accordingly.” 

“And  I know  why  maximum  deviation  didn’t  work  out,”  says  another.  “The  deviation  doesn’t  matter 
as  much  if  it  is  above  the  glideslope  as  if  it  is  below.  You  should  take  deviation  direction  into  account.” 

“Your  time  in  tolerance  looks  like  it  might  be  OK  if  you  would  just  change  the  tolerance  value  to  be 
more  in  line  with  the  way  our  good  pilots  actually  perform.  And  maybe  the  tolerance  should  be 
variable  - perhaps  a function  of  altitude  - because  tighter  control  is  critical  as  you  near  the  threshold.” 

Well,  no  wonder  things  looked  so  bad  on  the  initial  study!  As  a result  of  this  first  iteration,  we  might 
be  well  advised  to  plot  some  of  the  actual  performances  and  reconsider  the  problem  altogether.  We  could 
discover,  for  instance,  that  one  thing  unique  about  the  ;5  least  experienced  pilots  for  whom  we  have  some 
data  is  that  they  oscillate  about  the  glideslope  considerably  more  than  the  skilled  performers  do.  (Maybe 
the  number  of  glideslope  crossings  would  be  a good  measure!)  We  might  also  observe  that  the  more 
experienced  pilots,  when  they  do  deviate  significantly  from  the  glideslope,  make  very  gradual  corrections, 
whereas  the  novice  performers  tend  to  correct  more  rapidly,  and  they  often  overshoot.  (Maybe  rate  of  error 
correction  would  work!)  Finally,  we  may  see  that  the  good  pilots  (except  for  3)  never  descend  below  the 
glideslope,  even  though  deviations  above  it  are  sometimes  rather  large,  (Maybe  whether  or  not  descents 

below-.the  glideslope  occur  at  all  would  provide  at  least  part  of  the  answer or  might  this  just  be  a 

characteristic  of  a cautious  p3ot?) 

At  this • point,  our  original  list  of  3 potential- measures  has  tripled.  We  have  now  identified-the 
/ following?  measures  for  investigation: 


1 , RMS  glideslope  deviation  - unweighted 


Distance  from  Runway  (feet) 

re  J.  Hypothetical  perform* 


2.  RMS  glideslope  deviation  - weighted  by  altitude 

3.  Maximum  gjidcslope  deviation  - unweighted 

4.  Maximum  gHdesI ope  deviation  - weighted  by  direction  of  deviation 

5.  Time  in  tolerance  (Tol.  = ± A) 

6.  Time  in  tolerance  (Tol.  = f (altitude)) 

7.  No.  of  glideslope  crossings 

8.  Rate  of  error  correction 

9.  Whether  or  not  descents  below  the  glideslope  occur 

Iterations  2 through  K of  the  study  would  be  similar  in  nature  to  that  described  above,  and  typically, 
the  number  of  candidate  measures  would  vary  and  grow  multiplicatively  in  direct  proportion  to  the  number 
of  iterations  we  are  able  or  willing  to  conduct.  The  concluding  actions  of  the  study,  again  if  typical,  would 
be  one  or  more  of  the  following: 


(a)  Documentation  of  the  work  petformed  and  a recommendation  that  further  study  be  conducted. 

(b)  Determined  selection  and  use  of  a few  of  the  best-looking  measures,  with  reluctant  acceptance  of 
the  fact  that  they  lack  sensitivity  and  reliability  (but  by  golly  they  have  to  work  because  they  have  content 
validity!) 

(c)  Reconsider  whether  we  really  need  performance  measurement  techniques  at  all. 

(d)  Use  some  other  method  of  assessing  performance,  perhaps  one  that  seemed  to  work  OK  in  the 
1940’s  or  ‘50’s  (although  if  it  really  did  work,  there  would  have  been  no  need  for  this  ground  controlled 
approach  (GCA)  study  isi  the  fust  place!). 

The  purpose  of  the  preceding  tale  was  twofoid.  First  (although  certainly  not  enlightening  to  the 
readers  experienced  in  this  area),  it  illustrates  on  a comprehensible  scale  some  of  the  complexities  and 
problems  inherent  in  measurement  work,  at  least  as  it  is  commonly  approached.  To  put  it  amply,  the 
researcher  is  faced  with  assessing  the  performance  of  the  most  incredibly  complex  “black  box”  conceivable, 
and  many  times  without  even  the  benefit  of  knowing  the  standards  that  should  be  expected  of  it  as 
distinguished  from  those  that  appear,  in  practice,  to  be  expected  of  it.  (Certainly,  much  progress  lias  been 
made  in  rendering  the  human  black  box  white;  however  considerably  more  is  required  before  measurement 
of  human  performance  on  real- world  tasks  can  be  considered  straightforward.) 

The  second  purpose  was  to  lay  the  groundwork  for  a description  of  the  concepts  underlying  the  work 
reported  herein. 

Why  attempt  to  identify  and  laboriously  investigate  a few  hand-selected  candidate  measures, 
repeating  the  process  for  each  new  measurement  problem,  and  never  knowing  whether  or  not  the  measures 
most  suitable  have  been  simply  overlooked  or  unconccived?  Why  not,  instead,  define  a “universal” 
measure-set  which  encompasses  at  least  the  characteristics  represented  by  the  -o-caBed  classical  measures 
(and  then  some),  and  assign  to  computers  the  task  which  are  logically  theirs;  i.e.,  information  search  and 
retrieval?  In  other  terms,  what  is  suggested  is  that  a measure-set  be  designed  which  is,  in  effect,  inclusive  of 
measures  we  typically  select  for  investigation,  and,  moreover,  contains  the  power  to  generate  a myriad  of 
other  potential  measures  which  have  either  not  yet  been  conceived  and/or  are  too  numerous  to  list  for 
purposes  of  hand-selecting  those  that'  seem  appealing.  This  is  feasible,  and  the  measures  in  such  a set  are 
reasonable  to  investigate  if  the  characteristics  of  the  set  are  defined  rationally. 


II.  APPROACH 

The  approach  is  to  develop  a trial  measure-set  encompassing  characteristics  common  to  many  of  the 
classical  measures;  and  to  develop  a computer  program  which  generates  candidate  measures  from  the  set, 
executes  various  empirical  validation  tests,  and  prints  results  for  analysis.' 
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Measure  Set  Summary 

The  devised  measure  set  is  partitioned  into  three  subsets,  each  of  which  represents  measures  with 
different  characteristics.  One  subset  generates  candidate  measures  which  assess  performance  as 
characterized  by  unique  patterns  of  performance  variables  and  their  frequency  characteristics  (“Absolute” 
measures).  The  second  generates  measures  that  assess  performance  as  characterized  by  simultaneous  (or 
non-simultaneous)  occurrence  of  unique  events  (“Relative”  measures).  The  third  generates  measures  that 
assess  performance  as  characterized  by  unique  successions  of  events  or  system  states  (“State  Transfer” 
measures)  and  deviations  from  standards  (state  frequency  measures),  where  the  standards  are  either  defined 
by  the  user  or  computed  from  user-provided  performance  data. 

Introductory  Example 

In  way  of  example,  each  of  the  above  mentioned  types  of  measures  will  be  illustrated,  using  where 
posable  the  previous  example  of  a GCA  approach.  Air  attempt  will  be  made  to  demonstrate  that  the  three 
types  of  measures  comprising  the  defined  set  not  only  encompass  the  specific  measures  of  the  previous 
example,  but  conceivably  most  other  measures  commonly  (or -uncommonly)  selected  for  pursuit  in 
measurement  efforts  as  well  as  a host  of  previously  untried  ones. 

First  it  is  necessary  to  mention  (with  details  presented  later  in  the  report)  that  the  measures 
computed  are  based  on  a discrete  representation  of  die  performance  data,  derived  through  a transformation 
process.  -The  transformation  results  in  a representation  of  the  data  in  terms  of  the  number  of  units  by  which 
the  value  of  each  variable  (e.g.,  roll,  pitch,  altitude)  is  displaced  from  some  reference  level  or  reference 
function.  (The  size  of  the  unit-displacements  is  determined  partly  as  a function  of  performance  range  and 
variance.)  Thus  altitude,  in  the  GCA  example,  may  be  represented  by  several  Boolean  functions,  each  of 
which  denotes  whether  or  not  altitude  lies  in  a specific  band  around  the  glideslope  (e.g.;  a band  30'  wide 
located  1 00'  above  (or  below)  the  glideslope  may  be  represented  by  one  Boolean  function).  : 

States  of  the  (pBot/vehide)  system  are  represented  by  the  collective  states  of  the  various  Boolean 
functions  over  time  and,  in  turn,  are  represented  simply  by  numbers.  Thus  the  number  6 (binary  110), 
depending  on  the  Boolean  functions  being  investigated,  may  tell  us  that  at  that  sampling  instant,  the  pilots 
altitude  was  100'  £‘15'  above  the  glideslope  (first  binary  digit  (1)),  his  airspeed  was  120  knots  i 3 knots 
(second  binary  digit  (1)),  and  his  roB  angle  was  not  equal  to  zero  ± 2°  (3rd  binary  digit  (0)). 

It  is  this  state  representation  which  allows  us  to  efficiently  generate  and  test  measures  of  the  3 types 
described.  Any  measures  of  deviation  from  the  reference  function  (including  time  in  tolerance,  for  instance) 
are  inherent  in  the  collective  frequencies  with  which  the  various  defined  states  are  acquired  in  performance 
of  the  maneuver.  Any  measures  of  error  correction  or  it*  rate  are  inherent  in  the  transfers  that  occur 
between  various  states  over  time.  Measures  of  frequency  content  of  the  data  (inck'ding  the  number  of 
glideslope  crossings  in  the  previous  example)  are  inherent  in  the  state  transfers  that  occur  and/or  in  the 
“absolute”  type  of  measure  that  it  investigated.  Finally,  measures  which  relate  various  key  events  (e.g.,! 
smaller  glideslope  deviations  at  lower  altitudes)  are  inherent  In  the  “relative”  type  of  measure  that  is 
explored. 

Consider,  first,  the  measures  of  RMS  glideslope  deviation  in  the  previous  GCA  example. 
Math  -matically,  this  is  represented  as 


RMS=  Jn  i (Xi-Y,)1 

V M 

where  X is  the  actual  altitude  and  Y the  glideslope  altijude.  Equivalently  this  relationship  may  be 

conelatively  represented  by  mean  square  3 RMS3  4 - ( Y"  f.D,3) ' 

n 1=1 

where  Dj  are  specific  deviations  from  the  gjldestape  and  f.  are  the  frequencies  with  which  fee 
associated  deviations  are  encountered.  Similarly,  a weighted  mean  square  error  would  be 

RMS3  = — ( ^ W.  Fl  Dj3);  where  W.  are  Weights  assigned  to  each  deviation.  In  the  measure-subset  based 
n i*l 
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<Mi  state  frequencies  and  state  transitions,  the  D.  are  represented  by  the  number  of  units  displacement  from 
the  gEdeslope  associated  with  the  Boolean  functions  representing  altitude.  The  F}  values  are  the  computed 
state  frequencies  depicting  the  number  of  observed  samples  in  which  the  respective  Boolean  function  is  true 
(1). 


Since  the  referenced  measure-subset  encompasses  Fj,  D{,  and  related  vectors,  it  may  be  viewed  as  a 
vector  set  which  spans  the  space  of  measures  represented  by  RMS  glideslope  deviations,  weighted  or 
unweighted,  in  addition  to  any  other  conceivable  measures  attending  to  deviations  from  a reference 
function  (i.e.,  average  deviation,  maximum  deviation,  integrated  error,  etc).  Therefore,  the  referenced 
measure-subset  contains  a basis  for  the  space  of  measures  of  this  type,  and  any  measure  of  the  type 
mentioned  may  be  approximated  by  some  linear  combination  of  the  vectors  of  the  defined  subset.  It  is 
maintained  that  since  this  is  true,  all  measures  of  the  type  spanned  by  this  subset  may  be  explored  by  a 
computational  mechanism  which  can  generate  the  defined  vectors  and  perform  multiple  regression  analyses. 


Next,  consider  the  time  in  tolerance  measures  and  the  measure  of  whether  or  not  descents  below  the 
glideslope  occur.  The  former  is  represented  simply  by  the  frequency  of  occurrence  or  one  or  more  states, 
formed  by  Boolean  functions  involving  the  desired  tolerance  value.  A variable  tolerance  value  is 
automatically  included  in  the  analysis  because  the  different  Boolean  functions  themselves  (and  associated 
states)  represent  different  tolerances.  The  latter  is  also  represented  by  state  frequencies,  in  that  states 
representing  descents  below  the  glideslope  would  aB  be  zero  if  no  descents  below  tire  glideslope  occurred, 
and  non-zero  otherwise.  Therefore,  these  two  types  of  measures  are  included  in  the  computational 
mechanism  that  explores  the  measure-subset  based  on  state  frequencies  and  state  transitions. 


Finally,  consider  the  measures  (in  the  GCA  example)  of  number  of  glideslope  crossings  and  rate  of 
error  correction.  The  first  is  represented  by  the  number  of  transitions  that  occur  between  states 
corresponding  to  aircraft  positions  below  and  above  the  gideslope.  The  second  is  also  represented  by  state 
transitions  which;  (1)  distinguish  error  growth  from  decay  by  the  identity  of  the  states  between  which 
transitions  are  occurring,  and  (2)  assess  the  rate  of  growth  or  decay  by  the  relative  frequencies  of 
between-state  transitions  and  within-state  transitions.  Therefore,  these  types  of  measures,  too,  are  included 
in  the  state  frequency  and  state  transition  measure  subset  and  related  computatlonai  mechanism. 


This  single  measure-subset  therefore  covers  all  of  the  specific  measures  “selected”  in  the  previous 
GCA  example  and  much  more  - it  covers  the  general  types  of  measures  that  are  suggested  by  any 
considerations  of  deviation  from  a reference  function,  steady-state  or  transitive  positions  with  respect  to  it, 
and  movement  or  rate  of  movement  toward  or  away  from  it.  The  potential  power,  flexibility,  and  utility  of 
a computational  mechanism  exploring  this  variety  of  measures  is  significant.  A recent  unique  application  Of 
the  state  transition  concept  in  measurement  and  analysis  of  performance  is  described  in  Connelly  and 
Loental  (1974). 


We  have  yet  to  discuss  the  other  two  measure-subsets  (“Absolute”  and  “Relative”).  The  “Absolute” 
subset  and  its  respective  computational  mechanism  assesses  performance  characteristics  related  to  the 
repetative  frequencies,  periodicities,  and  associated  patterns  of  changes  between  various  states.  This  is 
accomplished  in  an  overall  manner  similar  to  that  described  previously;  i.e.,  vectors  which  span  these  types 
of  measures  are  generated  and  various  measures  are  explored  using  regression  analyses.  Examples  of 
measures  that  would  be  included  here  are  the  extent  and  type  of  “control  diddle”  used  by  an  operator; 
frequency  characteristics  of  an  operator’s  ballistic  response  to,  say,  a step  input;  and  measures  related  to  the 
number  of  control  reversals  used  in  performing  a segment  of  some  task. 


The  “relative”  subset  and  computational  mechanism  fills  an  identifiable  void  in  the  system  as  thus  far 
described.  If  takes  into  consideration  the  proximity  in  time  with  which  various  events  take  place  and  the 
conditional  probabilities  of  certain  events  occurring,  giver,  that  others  have  occurred.  Again,  the  approach  is 
to  generate  vectors  which  span  these  types  of  measures  and  employ  regression  analysis.  Examples  of 
measures  thereby  addressed  are  whether  or  not  e pilot  achieves  and  maintains  straight  and  level  flight 
whenever  he  is  within  a specified  distance  fium  the  threshold  on  a GCA  approach;  whether  or  not  he  begins 


to  roll  at  the  same  time  (not  before  or  after)  he  acquires  maximum  pitch  in  an  aerobatic  maneuver;  and 
whether  or  not  he  characteristically  achieves  a specific  (criterion)  airspeed  at  key  points  in  performance  of, 
say,  a lazy  8 maneuver. 

Summary 

Hie  preceding  subsections  describe  the  fundamental  concepts  of  the  approach.  The  intent  is  to  define 
a “universal”  set  of  measures,  not  by  enumerating  all  measures  in  the  set,  but  by  defining  their 
characteristics.  Three  major  characteristics  have  been  defined,  and  the  evolved  measure  set  is  represented 
accordingly  by  three  subsets.  It  has  been  shown  that  measures  generated  from  these  various  subsets 
encompass  a host  of  typically  selected  measures  (eg.,  those  of  the  GCA  example)  as  well  as  many  others 
possessing  the  subset  characteristics.  Mathematically,  this  may  be  viewed  as  developing  vectors  which  span 
various  measure-spaces,  and  it  is  proposed  that  the  measures  thereby  spanned  may  be  explored  using 
multiple  regression  analysis. 


IQ.  BACKGROUND  AND  STUDY  OBJECTIVES 

Many  of  the  basic  concepts  and  mathematical  techniques  fundamental  to  this  study  were  explored  on 
a trial  basis  in  previous  feasibility  studies  and  are  documented  in  the  references  (Connelly,  Schuler,  & 
Knoop,  1969;  Connelly,  Schuler,  Bourne,  & Knoop,  1971).  However,  efficient  computer  techniques  for 
exploring  the  various  types  of  measure-subsets  were  never  fully  developed  in  previous  efforts;  and  the  data 
transformation  techniques  as  well  as  the  measure  subsets  themselves  have  been  altered  and  refined  for  this 
study  on  the  basis  of  earlier  experience  vith  the  approach. 

The  purpose  of  the  present  study  was  originally  to:  (1)  refine  the  previously  explored  technqiues,  (2) 
develop  efficient  computer  implementation  methods,  (3)  validate  and  demonstrate  performance  of  the 
software,  and  (4)  apply  the  techniques  thereby  implemented  to  derive  and  validate  performance  measures 
for  five  training  maneuvers  flown  in  T-37B  aircraft  as  part  of  the  Air  Force  UPT  program.  Due  to 
non-technical  difficulties  encountered  in  collecting  the  required  student  and  instructor-pilot  data,  part  4 of 
the  original  objectives  had  to  be  abandoned,  and  the  objective  substituted  in  its  place  was  to  implement  and 
demonstrate  the  developed  software  on  the  Simulation  and  Training  Advanced  Research  System  (STARS). 
(The  STARS  system  is  located  at  the  Advanced  Systems  Division,  Air  Force  Human  Resources  Laboratory 
(AFSQ  Wright-Patterson  Air  Force  Base,  Ohio.  The  associated  digital  computer  is  a Xerox  Data  Systems 
(XDS)  Sigma  5.)  Therefore,  this  report  documents  the  computer  software  developed  and  the  related 
computational  algorithms  implemented  for  exploring  selected  types  of  measure-subsets;  however,  since  only 
a very  small  amount  of  data  was  able  to  be  collected  for  the  study,  it  was  not  possible  to  develop  and 
validate  any  specific  measures.  The  extensive  data  collection  and  reduction  machinery  developed  for  use 
(but  unfortunately  not  applied  in  this  study)  is  described  in  Knoop  and  Welde  (1973)  and  Gregory  and 
Cavanagh  (1973). 

Scope  of  Study 

The  study  includes  the  development  and  implementation  of  3 different  computational  mechanisms 
for  generating  candidate  measures  from  the  defined  subsets.  These  are  the  relative,  absolute,  and  state 
transfer  measures  previously  discussed.  A separate  computational  mechanism  for  state  frequency  measures 
was  not  included,  partly  because  the  state  transfer  mechanism  itself  generates  the  state  frequency  data  that 
is  needed.  Original  plans  were  to  develop  and  independently  test  a separate  state  frequency  mechanism 
using  this  generated  data  and  then,  depending  on  results,  interface  it  with  the  other  elements  of  the 
processor.  Due  to  the  previously  mentioned  change  in  program  objectives  and  associated  lack  of 
performance  data,  however,  this  was  not  able  to  be  pursued  beyond  the  planning  stage.  Emphasis  in  the 
study,  therefore,  was  on  developing  and  implementing  efficient  computer  techniques  for  the  3 developed 
computational  mechanisms  and  the  overall  computer-aided  processor  as  described  next. 
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IV.  COMPUTER-AIDED  PROCESSING  TECHNIQUES  AND  SUBSYSTEMS 


The  computer-aided  automated  pilot  performance  .measurement  processor  is  a*  FORTRAN  IV 
program  which  generates  candidate  performance  measures  through  various  operations  on  actual 
performance  data.  Thtsse  operations  indude:  ' ‘ • 


•a.  Develop perfofrmw.ee criteria, 

b.  Beterm&e ' he  ignifiewee  of  deviations  from  criteria, 

c.  Transform  ssjupit  pSKonBanie  d*ita  into  a compact  form  for  processing, 

<!,  Coaossct  a catena struTiwahied  search  for  candidate  performance  measures, 

e.  Perform  validation  i&te,  and 

f.  Provide  data  management  processes. 

A generalized  flow  diagram  of  the  processor  appears  in  Figure  2. 


A primary  task  in  developing  performance  measures  is  the  determination  of  standards  or  reference 
functions.  Performance  standards  should  define  the  unique  manner  in  which  the  operator  should  perform 
the,  task.  Often,  however,  there  are  a number  of  satisfactory  ways  to  properly  accomplish  a task  and  there 
may  exist  a family  of  reference  functions  representing  criterion  performance.  As  a result,  the  reference 
function  forms  employed  by  the  processor  may  accept  parameters  provided  by  the  user  or  estimated  from 
sample  performance  dat*. 


Multi-variable  regression  is  used  in  formulation  of  reference  functions  from  sample  data.  The  idea  is 
to  extract  from  demonstrations  of  superior  performance  functions  which  uniquely  represent  that 
performance.  Evaluation  of  the  function  fit  is  accomplished  through  analysis  of  residues.  A small  residue 
value  indicates  a convenient  clustering  of  all  superior  performance  data,  while  a large  residue  value  indicates 
that  the  regression  formulation  is  not  appropriate  or  that  other  parameters  are  required. 


An  additional  test  of  the  candidate  functions  is  made  by  comparing  reridues  obtained,  from  the 
superior  performance  category  data  with  those  obtained  from  other  performance  category  data  such  as 
good,  fan,  and  poor.  The  difference  between  the  residues  obtained  is  an  indication  of  the  potential 
performance  discrimination  capability  of  a measure  developed  from  that  criterion. 

A second  important  step  in  the  development  of  performance  metrics  is  the  determination  of  the 
relevance  of  deviations  from  the  reference  performance.  It  should  be  noted  that  the  importance  of  operator 
errors  is  generally  not  constant  over  the  entire  problem  state  space.  Thus-,  some  systematic  means  must  be 
provided  to  test  various  types  of  deviations  and  patterns  of  deviations  as  to  their  relevance  to  performance 
measurement.  Table  1 shows  various  ways  that  deviations  from  the  criterion  or  reference  might  be  related 
to  performance  measurement  The  processor’s  capability  to  assess  flic  significance  of  a wide  variety  of 
relationships  such  as  these  is  automatically  assured  due  to  the  types  of  performance  measures  it  is  designed 
togenerate and  test. 


The  processor  has  four  main  portions: 

1.  Input  and  preparation  of  data,  including 

a.  Data  management 

b.  Smoothing 

c.  Maneuver  Sectoring 

2.  Generation  of  criterion  functions  via  regression  analysis 

3.  Processing  of  data  by  adaptive  mathematical  models 

4.  Testing  and  specification  of  performance  measures. 
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I'nblf  I Some  Possible  Criterion  and  Performance  Measure  Factors 


Type  or  Criterion 


Possible  Ways  Deviation  (Error) 
It  Related  to  Performance 


Functions  Relating  Piohlcin 
Variables  (Reference  Path) 


Differential  Reference  (whc.e  criterion 
is  specified  by  differential  or  difference 
equations) 


Fixed  (variable)  tolerance  at  a specific 
time  or  at  a specific  value  of  another 
variable 

Sequence  of  Operation 


< ■ Amount  of  deviation  from  path 

<■  Max  deviation 

< * Time  in  a tolerance  band 

< > Convcrgencc/divergence 

<•  Similarity  to  reference  path 

<■  Shape  of  deviation 

<>  Time  significant  deviation  occurs 

Frequency  of  significant  deviations 
! • Rate  of  error  correction 

( ' Way  error  is  corrected 

< ■ Number  of  errors  that  occur  simultaneously 

< • Error  in  differential 

<>  Critical  variable  values  exceeded 

f>  Time  critical  variables  values  are  exceeded 
° Convcrgencc/divergence  to  reference  point  on 
path  trajectories 
o Shape  of  trajectory 

<>  Variable  out  of  tolerance 

o Amount  variable  is  out  of  tolerance 

° Time  variable  is  out  of  tolerance 

o Number  of  errors  in  sequence 

° Number  of  critical  errors  in  sequence 


Data  Management 

Due  to  the  great  volume  of  data  that  must  be  handled  by  the  processor,  systematic  data  management 
is  of  great  importance.  This  is  basically  a housekeeping  operation  which  controls  the  coding  of  data  and  its 
efficient  storage  and  retrieval. 

Data  Smoothing 

Examination  of  recorded  flight  data  shows  occasional  noise  “glitches”  on  the  data  samples.  These 
glitches  occur  at  random  times  and  must  be  removed  prior  to  processing.  Noise  glitches  are  assumed  to  be 
pulses  applied  to  the  filters  that  smooth  data  prior  to  sampling.  Thus,  the  noise  pulse  appears  as  a pulse 
with  an  exponential  decay  as  shown  in  Figure  3.  The  resulting  sampled  values  show  a !argc  sample  to  sample 
delta  change  between  the  samples  before  and  after  the  noise  pulse. 

Detection  of  the  noise  is  accomplished  by  comparing  the  sample  to  sample  (delta)  change  with  a 
pre-established  criterion  value  as  follows: 

I aj+,  ad  < c,  i = 1.2. . . . n (I) 

where  aj  is  a sample  value  and  c is  a delta  criterion  value.  If  the  inequality  is  not  satisfied,  a noi.»c  pulse  is 
assumed  to  exist. 

Once  a noise  pulse  is  delected,  the  time  duration  of  the  disturbance  must  be  determined.  Expciienec 
has  shown  that  the  nominal  disturbance  duration  can  he  expected  to  he  .1  seconds  (10  samples  at  a 
sampling  rate  of  100/scc.)  for  the  recorded  T-37  flight  data.  The  duration  of  the  disturbance  is  computed 
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by  examining  the  value  of  future  samples  one-by-one  to  determine  which  sample  value  first  falls  within  the 
acceptable  region.  The  test  is  given  by  the  inequality: 


lai+j  " ai 1 <c*j>  i= M 


(2) 


The  smallest  value  of  j for  which  the  above  inequality  is  true,  is  one  greater  than  the  number  of  *»mples 
included  in  the  glitch. 

When  the  samples  included  in  the  noise  glitch  have  been  determined,  those  ample. values  are  replaced 
by  interpolated  values.  These  values  are  determined  by: 


a...  = a-  + 
»+>  » 


(3) 


where  n it  the  value.of  j for  which  inequality  2 is  true. 

Maneuver  Sectoring 

a.  Introduction.  Automatic  performance  assessment  normally  involves  the  computation  of  several 
different  measures,  each  of  which  corresponds  to  different  aspects  of  (or  skills  used  in)  the  task  of  interest. 
In  most  flight  maneuvers,  the  skills  required  (and  measured)  vary  from  segment  to  segment  of  the 
maneuver.  For  example,  different  skills  (and  measures)  are  required  during  the  downwind  portion  of  an 
approach  and  landing  than  are  required  during  the  turn  to  final  or  during  the  flare  and  touchdown. 
Therefore,  it  is  necessary  to  segment  the  maneuver  by  identifying  natural  breakpoints  which  delineate  . 
portions  requiring  computation  of  different  measures.  Once  these  breakpoints  have  been  identified, 
algorithms  and  computer  techniques  are  needed  for  automatically  detecting  them  on  the  basis  of  recorded  ■ 
pilot  performance  data.  This  section  describes  the  development  of  such  techniques  for  inclusion  in  the 
processor  and  their  trial  application  to  several  undergraduate  pilot  training  flight  maneuvers.  It  is 
appropriate  to  point  out  that  automatic  segmentation  of  performance  also  has  utility  in  a number  of 
advanced  simulator  training  capabilities.  For  example,  an  increasing  number  of  requirements  for  and 
appfications  of  automatic  malfunction  insertion  are  emerging  in  recent  and  current  flight  simulator 
developments.  To  automatically  insert  a malfunction  at  the  point  in  a mission  that  is  realistic  for  the 
malfunction  and  at  which  the  highest  training  value  is  expected,  it  is  first  necessary  to  automatically  detect 
the  desired  point  (thus,  the  utility  of  automatic  segmentation).  Other  advanced  training  capabilities  such  as 
reinitialization  of  the  simulator  and  subsequent  playback  of  a portion  of  the  performance  also  can  make  use 
of  segmentation  techniques  (for  automatically  detecting  the  point  from  which  playback  is  desired).  Finally, 
the  distinct  trend  toward  the  use  of  cathode  ray  tube  (CRT)  displays  (rather  than  or  in  addition  to  aircraft 
repeater  instruments)  at  simulator  instructor  stations  suggests  another  application  of  automatic 
segmentation.  Present  display  techniques  are  to:  (1)  always  display  everything  the  instructor/operator  may 
ever  need  to  see  during  the  entire  mission,  or  (2)  allow  various  CRT  “pages”  to  be  manually  selected.  The 
first  technique  is  objectionable  due  to  the  number  of  displays  required  and  subsequent  load  on  the 
instructor/operator  information  sorting  and  processing  requirements.  The  second  is  equally  objectionable 
due  to  the  instructor/operetor  information  retrieval  load.  Automatic  segmentation  techniques  could  be 
used  to  assure  that  display  contents  always  suit  the  instructor/operator  needs  based  on  what  the  student  is 
practicing.  The  techniques  described  herein  could  be  usefully  employed  for  any  of  the  above  applications. 

b.  Approach.  The'  approach  was  to  develop  techniques  for  generating  a mathematical  representation 
of  the  state  of  pilot/aircraft  performance  which  could  be  applied  to  any  maneuver.  Using  this 
state-representation,  segmentation  logic  was  developed  for  detecting  specific  states  corresponding  to  the 
desired  breakpoints  within  each  maneuver.  (The  breakpoints  themselves  were  identified  largely  on  the  basis 
of  maneuver  analyses  performed  as  a part  of  other  performance  measurement  studies  (Connelly,  Bourne, 
Loental,  MigSaccio,  Burchick,  A Knoop,  1974).  This  section  describes  the  state  representation  techniques 
and  the  baric  segmentation  k>gc  that  was  developed. 

(1)  Maneuver  State  Representation 

the  technique  for  representing  maneuver-performance  states  was  to  model  significant  aspects  of  the 
various  performances  using  Boolean  functions.  The  specific  Boolean  functions  used  differed  from  maneuver 
to  maneuver  as-applied  in  various  combinations  to  represent  desired  states.  However,  since  many  of  the 
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functions  were  found  to  be  applicable  to  more  than  one  maneuver,  the  approach  used  was  to  develop  a 
universal  function-set  [ and- associated  Boolean  notation  which  could  be  computer-implemented  and, 
collectively,  would  satisfy  state-representation  requirements  for  all  the  maneuvers. 

The  devised  Boolean  functions  and  the  associated  notations  included  indications  that  a specific 
condition  (Boolean  Function)  is  presently  true  or  that  the  condition  was  previously  true  at  least  once. 
Furthermore,  functions  and  notations  were  developed  to  indicate  when  a condition  is  first  true,  last  true 
(i.e„  when  it  becomes  false),  and  the  interval  of  time  during  which  the  condition  is  true. 

The  notation  used  for  representation  of  states  is  summarized  in  Table  2.  As  shown  in  the  table,  the 
Boolean  notation  “A  = 1”  is  used  to  indicate  that  condition^  A is  presently  true,  and  “A  = 0”  is  used  to 
indicate  that  condition  A is  presently  false.  In  addition,  “A”  indicates  whether  or  not  A has  been. true 
during  the  maneuver.  Initially,  A is  set  equal  to  zero;  if  condition  A becomes  true  at  least  once  during  the 
maneuver,  then  A is  set  equal  to  1 for  the  remainder  of  that  maneuver.  This  provides  Boolean  notation  with 
a ‘'memory”  and  allows  a lope  function  to  be  written  in  terms  of  present,  as  well  as,  previous  events. 


Table  2.  State  Representation  Notation 


Notation 

A=  1 

Condition  A is  presently  true. 

A = 0 

Condition  A is  presently  false. 

x=o 

Condition  A is  not  and  has  not  been  true 
during  this  maneuver. 

X=l 

Condition  A is,  or  has  been  true  during  this 
maneuver. 

t(A) 

Time  A became  true. 

i(X) 

Time  A first  became  true. 

t(X) 

Time  A became  false. 

{ A=X<z} 

Defines  logic  variable  as 
A=l'ifX<Z 
A = 0 if  X > Z 

The  time  that  events  take  place  is  also  important.  Thus,  t(A)  represents  the  time  that  condition  A 
became  true,  and  t(A)  is  the  time  that  condition  A became  false.  This  is  illustrated  in  Figure  4 where 
condition  A is  true  for  a period  of  time  and  then  false.  Note  that  the  symbol  t(A)  indicates  the  time  that 
condition  A first  became  true  and  is  always  equal  to  some  corresponding  t(A).  However  t(A)  itself  may  vary 
over  ths  maneuver  if  the  associated  condition  (A)  changes  from  false  to  true  more  than  once.  , 

. The  deviaed  functions  and  notation  can  be  used  to  detect  the  sequence  in  which  events  occur. 
Consider  the  time  plots  of  two  aircraft  variables,  pitch  (0)  and  roll  ($>),  shown  in  Figure  4.  In  plot  1 in  the 
figure,  fitch  reaches  zero  first;  whereas  In  plot  2 roll  reaches  zero  first.  In  cases  1 and  3y  accordingly,  the 
^leiri  condfeiohs^  B and X indicate  wheri-tlie  tw>  variables  of  interest  are  zero.  In. case  2 and  4,  the 
B«)i^  c^fioiu  with  “memory”  (B  uid  t)  remain  true  for  the  subsequent  time  samples  after  they  first 
become  true.  (In  cases  5 through  8 and  9 through  12,  AND/OR  combinations  of  the  Boolean  variables  are 
iflustrated,  respectively.) 


Tte  Aotafiqn  provided*  concise  and  easily  applied  framework  within  which  all  desired  statescould 
be  and  tested.  Although  simple  in  appearance  (Table  2),  the  notation  is 

powerful  enough  for  uae  in  developing -logic  which  detects  the.nature  as  well  as  fire  sequence  of  various 
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Condition  A 


Case 

1: 

B 

ii 

"S' 

It 

o 

Case 

2: 

B 

Case 

3: 

C 

o 

1! 

ta- 

ll 

Case 

4: 

C 

Case  5:  B*C 

Case  6:  B • C 
Case  7:  B*C 
Case  8:  B*C 


pilot  actions,  as  observed  through  their  affect  on  aircraft  performance.  The  following  paragraphs  describe 
the  Boolean  function  set  that  was  developed  and  the  segmentation  lope  used  for  five  flight  maneuvers. 

(2)  Segmentation  Lope 

Table  3 lists  the  set  of  individual  Boolean  functions  identified  for  use  in  developing  the  segmentation 
lope.  The  function  numbers  shown  in  Table  3 are  used  in  Table  4 to  summarize  the  segmentation  lope 
developed  for  each  of  five  maneuvers.  Note  that  only  17  of  the  Boolean  functions  were  required  for  the  five 
maneuvers  that  were  investigated.  Based  on  this  observation  and  the  representativeness  of  the  maneuvers 
tested,  it  appears  that  the  Boolean  function  set  is  more  than  adequate  for  developing  segmentation  lope  for 
conceivably  any  basic  flight  training  maneuver. 

Tables  5 through  9 present  the  segmentation  lope  for  each  maneuver  separately,  including  a 
descriptive  title  of  each  segment  and  the  Boolean  functions  used  for  detecting  the  states  determining  the 
desired  breakpoints  between  segments. 

c.  Trial  Applications.  To  perform  initial  tests  of  the  lope,  sample  pilot  performance  data  recorded 
on  a T-37B  Preraft  were  used.  (The  bade  data  acquisition  system  is  described  in  (Knoop  & Welde,  1973). 
For  this  data  collection  effort,  some  revisions  were  incorporated  including  the  addition  of  stick  force 
sensors  and  the  increase  of  both  range  and  reliability  of  Preraft  attitude  sensing.  Revisions  made  to  the 
oripnP  data  acquisition  system  are  documented  in  (Gregory  & Cavanagh,  1973).)  The  performances  were 
flown  by  instructor  pilots  who  purposely  demonstrated  examples  of  how  a novice  might  perform  each 
maneuver  as  well  as  examples  of  skilled  maneuver  performance.  Four  flights  of  each  of  the  5 maneuver 
types  were  used,  two  of  which  were  rated  excellent  by  the  performing  pilot  and  two  rated  poor.  This 
provided  examples  of  both  performance  extremes  for  testing  the  segmentation  lope. 

(1)  Cloverleaf 

The  cloverleaf  maneuver  consists  of  a pattern  of  four  consecutive  loops,  or  leaves,  PI  identicP  except 
for  heading.  For  purposes  of  explanPien,  only  the  first  leaf  is  discussed . 

The  leaf  is  begun  after  the  start  condition  (level  flight)  is  satisfied.  Figure  5 is  the  computer  printout 
of  one  leaf  of  an  excellent  cloverleaf  as  processed  by  the  lope  (segmentation)  program.  Sector  2 bepns 
when  the  pilot  pitches  up  above  T2 . He  then  bepns  to  roll  (sector  3)  until  he  reachec  a maximum  roll  vPue 
(Cmj).  Although  excellent  pilots  generally  roll  to  180°,  poor  pilots  often  do  not  achieve  180°;  hence,  roll 
maximum  is  used  to  trigger  the  start  of  sector  4 because  the  lope  must  work  on  all  types  of  flights.  In 
sector  4,  the  pilot  rolls  back  and  pitches  down  until  he  reaches  a minimum  pitch  (Bm2)-  Most  pilots, 
regardless  of  thdr  proficiency,  begin  to  roll  out  before  a pitch  of  -90°  is  attPned;  therefore,  sector  5 
triggers  on  minimum  pitch.  The  pilot  levels  off  his  pitch  (sector  6)  prior  to  entering  the  next  leaf,  then 
bepns  the  leaf  by  pitching  up  again  (sector  2). 

(2)  Splits 

The  split  s is  an  evasive  type  maneuver  in  which  the  pilot  effects  a 180°  heading  change  by  pitching 
up,  rolling  over,  and  pulling  out.  The  plot  from  the  lope  program  is  shown  in  Figure  6.  Initially,  sector  5 
triggered  on  pitch  = -90°.  However,  as  with  the  cloverleaf,  most  pilots,  excellent  and  poor  alike,  roll  out  as 
they  pitch  down  and  never  reach  -90°.  Therefore  the  condition  was  changed  to  minimum  pitch  (®M2)* 

(3)  Lazy  8 

This  maneuver  consists  of  two  halves,  each  of  which  are  identical  except  for  heading  and  direction  of 
roll.  The  start  condition  (level  flight)  is  a function  of  pitch  and  roll,  while  the  subsequent  sectors  are 
identified  solely  on  pitch  angle.  Figure  7 shows  lope  program  output  for  the  first  half  (sectors  1-5)  of  a 
lazy  8. 

• (4)  Normal  Landing 

the  landing  maneuver  is  made  up  of  five  sectors.  A sample  lope  program  output  is  shown  in  Figure  8. 
On  the  sample  flights  examined  in  this  study,  the  pilot  did  not  land;  instead  he  performed  a touch  and  go 
maneuver.  In  either  case,  the  maneuver  is  logically  terminated  when  the  touch  down  condition  is  detected. 
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Table  3.  Boolean  Functions 


C,  ' 

!0  =0,  ±T,l 

»>T.l 

0<T3  J 
0 = +45°  ± T4  i 
{0  = —90°  ± Tj  | 

r 0a«4 

i ~ 9=flm4l 

*>Ilf 
*<h\ 

U = +180o±u} 

U=+90°±i,J 
^ = -90°  ±14 
l = ^ = 

1 ~ = 

- tAs<TasI| 

= |rs  = Neutral  ±r4' 


, are  tolerance  tacto: 


ciovtiiul 


lo.  Function 

Ej.=  {rs  = Full  left  } 

E3  = I Rs  = Full  right } 

E4  = |Rsj  Reversed^ 

*1^2  E3  + E3  Ej  } . 
F,  = IDt  = Neutral  ± S,  f 
F2  =1  Dt  = Full  forward  } 
Gj  = | T = Idle  J 
32  . Future  Expansion 

C, 

39  Future  Expansion 
C2  + C3 
B1  - Cl 
B2  + C2  + C3 
Bl+Cl 
CM1  • Cl 
CM2*  Cl 
CM3*  Cl 
C5  + C6 
E2  + E3 

Future  Expansion 
B1  * Cl  Stop  Condition 


l as  follows  for  the  five  maneuvers: 

Landliw  Barret  Roll 


sCoraputation  of  successive  maximum  or  minimum  values  such  as  $MX  ^ ®MX+1  requires  that 
an  intermediate  null  condition  (Bj)  be  true.  Thus  the  following  sequences  must  occur  in  order  for  successive 
extreme  values  to  be  etfabfahcd:  »MX.  Bj,#mx+1.  Bjs*MX+2>  • • * 

dMY.C|,dMy+i,Ci,dMY+2,... 

‘ *Rt  ’'.genrvsl  tohrsnce  on  rtidder  position 
Sj  '*  f«iu  rl  tofesance  or,  stick  position 

*0  - v'enb 

’■a  "an  * 

*M1>  dtsSt  - Ids  focal  rcarlma 
At  -sfoipced 

Ra  - rudder  ptskfcm 

• Dt  - longitudinal  stick  position 

T • tSiitktie  petition 


amVn:  J X».  J^iL:  >,  •>  - . - , 


Table  4.  Summaiy  of  SegmoiUtion  Logic 


'■  . ,'  SOgwtWti  ■ ' 

X 3 1 4 I 4 7 • •-  18  It  12  13  14  IS  14  17  14  14  20  31  22 


Clowrlcaf  1 0 2 40  17  7 .1  50 

- 2 * 2 40  17  7 1 50 

3 2 40  17  7 1 50 

4:  2 40  17  7 1 50 

Splits  1 0 2 40  3 7 50' 

Lazy  8 1 0 42  6 1 f 

2 43  8 1 9 50 

Nonna! 

Landing  1 0 40  -44  40  45  20 

Barrel  Roll  1 0 3 2"  47  43  50 


TabieS.  Maneuver  State  Logic  for  a Goverieaf 


CanOHIon 


OoolMtn 

flHWlM 


Function* 

Numfcor 


Entry 

Pitch  = 0°±T,  and 
R6B*0°±I,  ‘ 

B,  • C, 

0 

Climb 

Pitch  >T, 

Ba 

2 

Roll 

Rol  < I3  (Left)  or 
Ro1>12  (Right) 

C2+Cj 

40 

Pitch  to  90° 

Rbfl  4m, 

17 

PuBThm 

Pitch  *9M2 
Pitch  *o”±  T,  * 

BM2 

B, 

7 

RwJ  (entry  to 
next  leal) 

1 

•From  Table  3. 
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Table  6.  Maneuver  State  Logic  for  a Split  S 


Ssetor 

Numfear 

Sector 

Nam* 

Condition 

Boolean 

Function 

Function* 

Number 

1 

Start 

Pitch  =0°±T  and  B,  • C, 

0 

Roll  = 0°  A I, 

2 

Entry 

Pitch  >Ta 

B2 

2 

3 

Inversion 

Roll  > J2  or 
Roll  < I3 

c2  +c3 

40 

4 

Pull  thru  to  90° 

fitch  <T3 

B3 

3 

5 

Pull  thru  to  0° 

Pit^  = 5M2 

BM2 

7 

Stop 

fitch  = 0°  ± T, 
Roll  = 0°  ± lj  ' 

und  Bj  • Cj 

50 

♦From  Table  3. 

Table  Z 

i 

Maneuver  State  Logic  for  a Lazy  8 

Sector 

Number 

Sector 

Nam* 

Condition 

Boolean 

Function 

Function* 

Number 

1 

Entry 

fitch  = 0°±T,  and 
Roll  = 0°  ± L 

B,  - C, 

0 

2 

1st  Quarter 

fitch  > T2  or 
Roll  > I2  or 
Roll  < I3 

b2  +c2  +c3 

42 

3 

2nd  Quarter 

Pitch  = flMl 

®M1 

6 

4 

3rd  Quarter 

fitch  = 0°±T,  ' 

B, 

1 

5 

4th  Quarter 

fitch  = 0M2 

BM2 

7 

6 

1st  Quarter 

fitch  = 0°  ±T!  or 
Roll  =0°  ± lj  ' 

8,  + C, 

43 

7 

2nd  Quarter 

Pitch  = 0M3 

BM3 

8 

8 

3rd  Quarter 

fitch  = 0°±T, 

B, 

1 

9 

4th  Quarter 

fitch  = eM4 

BM4 

9 

Stop 

End 

Pitch  = 0°  ±TS  and 
RoC=0°iI,  ’ 

Bi-C, 

50 

♦From  Table  3. 
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iMtH 

Numbo* 


Tables.  Maneuver  State Logic (or a NonndLeoding 


Condition 


'-■ieleeii-v  -'■  "■ 

,-'  Function  . ■ •••  ■ \.  ■ 


1 

Start 

Pitch  = 0°  ±Tj  aind 

' B.  - C,  * ;, 

- 

. -v.  R6n=0°±J,  ‘ 

- \ ?>  Y,  1 • sr •■ 

x 'V  \i:  "S',  i , ,zSl  k . 

2 

i 

Pitch  Out 

RoD  > lj  or 
Ro#<  I3 

Cj  + Gj 

' '40 

! 3 

Downwind 

Rofl=0°±l, 

-.  after,  RoO  = $M1 

Cmi*6! 

• 44  -...  ;7.. ,;  \ 

4 

Final  Turn 

. RoU  > !j  or 

* G2+C3 

40  77V 

> 

v‘ 

Rq»  <4 

. J ' < ?'  ' 

. 5 

i 

t 

Find  Approach 

:RoB;*0o  ±lj 
after  Roll  = $m2 

9^2  *:P»:  . 

' - 45  ■ ■ " 

J • 7 .*  ' -1  *v 

Stop 

(Touch  Down  or 

Airspeed  <TASi 

D, 

-20  . :(Vf  ‘ 

J 

Touch  and  Go) 

- 

♦From  Table  3. 


sector 

Number 


Table  9.  Maneuver  State  Logic  for  a Barrel  Ro5 

Boolean 

Condition  Function 


Function* 
Number  • 


1 

Start 

Pitch =0°±T,  and  • 
RoU  =0°  ± Ij 

Bj  * Cj 

0 

2 

Entry 

Pitch  <T3 

B3 

•3 

3 

1st  Quarter 

Pitch  > Tj 

Bs 

2 

4 

2nd  Quarter 

Roll =99°  i Is  or 

RoB  = -90°  ± 4 ‘ 

Cs+Ce  ' 

47 

5 

3rd  Quarter 

Ron  = 180°±4 

c4 

14 

6 

4th  Quarter 

Roll  = -90°  ± 4 or 
Ron  = 90?±I,  . 

Cs+C6 

- 47 

7 

End 

Pitch  = 0°  ±:Tjdr 
Roll  *0°  ± 4 ’ 

Bj  +Ci  *" 

;■  >4  - 

Stop 

Pitch  -0°  ±7,  and 

Bj  *%  ' . 

50 

. t 

R6n*0°  ± l,  * 

• \ 

■ . ' 

:7« 

■ ;:.h 


vM&jjkSSi 


. 24  • 


' *s  y y* 

7 ■'  ' V* v 


I'  * ~ ^ *'  >:?<*£ 

^jy;s!r 


♦See  Table  3. 


Figure  5.  Logic  plot  of  one  leaf  of  a 
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Figure-7.  Logic  plot  oil  onc4wlf  of  • kzy;8, 
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Figure  8.  Logic  plot  for  a normal  landing. 


(5)  Baird  Roll 

This  is  an  acrobatic  maneuver  in  which  the  pilot  rolls  through  360°.  A sample  logic  program  plot  ' 
appears  in  Figure  9>The  sample  data  indicates  that  inexperienced  pilots  have  difficulty  attaining  level  flight 
(Bj  « C,)  at  the  end  of  the  4th  quarter  (sector  6);  consequently,  sector  6 contained  a long  tail  of  data  not 
representative  of-the  specific  pilot  task,  we  were  trying  to  isolate..  A seventh  sector  was  added  to  allow 
termination  of  sector  6 if  either  pitch  or  roll  is  level  (Bt  + Ct).‘ The  maneuver  terminates  when  the  pilot 
attains  level  flight. 

Regresdon  Analysis 

a.  Introduction.  A major  purpose  of  the  regression  analysis  is  to  generate  reference  functions  which 
are  representative  of  excellent  performances.  These  reference  functions  are  automatically  generated  by  the 
processor  for  use  in  deriving  performance  measures. 

A number  of  reference,  functions  are  constructed  for  each  sector  of  each  maneuver  type.  Each 
function  h a mathematical  representation  of  certain  parameter  relationships  characteristic  of  that  sector. 
Deviations  of  an  actual  flight  from  thk  function  are  computed.  A' standard  set  of  operations  on  these 
deviations  are  performed,  and  results  are  tested  for  performance  discrimination  content.  The  techniques  of 
measuring  and  interpreting  these  deviations  are  discussed  in  detail  in  later  sections  of  this  report. 

A .useful  reference  function  must  be  consistent,  in  that  it  produces  small  deviations  with  data  from 
excellent  performances;  at  the  same  time,  it  must  be  able  to  provide  discrimination  in  tests  among  various 
performance  levels.  In  the  processor,  reference  functions  are  generated,  then  tested  for  consistency  and 
discrimination  capability.  This  procedure  is  discussed  in  the  following  sections. 

b.  Theory . Several  reference  functions  are  generated  for  each  sector  of  each  maneuver  by  performing 
a least  squares  regression  analysis  on  selected  skilled  performance  data  for  the  specified  sector.  In  our  initial 
analysis,  four  candidate  reference  functions  are  generated  for  each  maneuver  sector  by  using  data  from  two 
available  excellent-rated  flights  of  each  maneuver  type.  The  technique  is  illustrated  in  Figure  10. 

(1)  Regression  Computation  Method 

In  applications,  many  sample  flights  of  several  performance  categories  will  be  used  to  form  the 
reference  functions  and  update  them.  The  data  are  initially  arranged  on  tape  by  maneuver  type  and  it  is  not 
feasible  to  store  all  data  or  to  reread  the  tape  for  each  maneuver  sector.  Therefore,  it  is  desirable  to  use  a 
technique  which  allows  updating  of  the  regression  coefficients  without  having  to  store  all  previous  raw  data. 
The  method,  discussed  in  detail  in  Connelly  et  al.,  (1969,  pp.  179-181),  represents  the  data  in  a compact 
summary  form.  Briefly,  the  problem  is  stated  as: 

N 

Y = A + £ B.X.  (3) 

1*1 

where  Y is  a factor  of  interest  (dependent  variable)  and  X,  is  a combination  of  the  system,  variables 
(independent  variables).  Given  T samples  or  experiments,  the  method  of  least  squares  minimizes  P: 

' T N 

2 (-VA+  2 BX/  (4) 

i=l  j*l 

and  gives  solution  values  for  the  coefficients: 

T T N T T T N 

-2  W X 2 Wh*  2 x'kitt n 2 Y,-1/T2  2 BjX  )=o 

i*i  i*i  j*i  • . i*i  . i*t  i«i  H 
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where 


K = l.N 

T T N 

A = I/T(2VJ  2 BjX.) 
i=1  i=l  j=l 

Defining  the  following  sums: 


(5) 

(6) 


SUMYX0C)  = S Y.  Xki 
i=l 


T 

__  <r.'3 


SUMXX(J,K)  = 2 XjiXki 
i=l 


(7) 


(8) 


SUMX(K)=£  Xk. 
i=l 


(9) 


SUMY  Y. 
i=l 

allows  representation  of  the  problem  as: 

N 


00) 


N 


0 = -SUMYX(K)  + 2 B.  SUMXX(J,  K)  + SUMY  * SUMX(K)/T  - SUMX(K)/T  * J B SUMX(J) 

j=l  ' j=l 


where 

K = l.N 

Solving  for  the  B’s  is  done  via  a matrix  approach: 


Let  B - 


0(1) 

0(2) 


L B(N), 


(10 


(12) 


R = 


SUMYX(1)  - SUMY  * SUMX(1)/T 
SUMYX(2)  - SUMY  * SUMX(2)/T 


L SUMYX(N)-  SUMY  * SUMX(N)/T  J 


(13) 
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Q(J,K)  - SUMXX(JtK.)  - SUMX(K)  * SUMX(J)fT 
(known  as  correlation  coefficients) 


(M) 


SUM  * 

“00.  ll 
0(1.2) 

0(2.1)  ... 

0(2,2)  ... 

O'*.  1)  - 
Q(N.  2) 

-0(1.  N) 

0(2.  N) 

Q(N,N)_ 

(an  NxN  matrix) 

Tlie  problem  is  now  written  as: 

SUM • B ~R 

and  the  solution  is: 

B - SUM1  • R 

Now  A can  be  computed  from  equation  6: 


N 

A = 1 /T  ^SUMY  - 2 B(J)  SUMX(I)  ^ 

For  ease  in  programming,  the  matrices  SUM  and  R can  be  further  broken  down: 

SUMXX(l.  I)  SUMXX(2,  1)  • • • SUMXX(N,  1)] 

SUM  XX  - . . 

SUMXX(1  ,N)  SUMXX(2,N)  • • • SUMXXfN,  N) 

SUMXJ  ~ [SUMX(1)  SUMX(2)  . . . SUMX(N)] 


SUMXK 


SUMX(t) 

SUMX(2) 


SUMX(N) 


Then  SUM  = SUMXX  - \ IT  * SUMXK  * SUMXJ 


Let 


rsuMYXd)' 


SUMYX 


l^UMYX(N)J 

Then  R * SUMYX  - SUMY/T  • SUMXK 


(15) 

(16) 
07) 

(18) 

(19) 

(20) 
(21) 

(22) 

(23) 

(24) 
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Therefore 

B = [SUMXX  - IfT  SUMXK  * SUMXJ  ]*»  • [ SVMYX  - SUMY/T  • SUMXK]  (25) 

This  technique  provides,  for  efficient  computer  usage  by  compact  storage  of  all  previous  data  that  is  needed 
for  updating  of  the  regression  coefficients. 

(2)  Candidate  Reference  Functions 

In  the  processor,  reference  functions  are  based  on  data  from  several  excellent  maneuvers.  Initially, 
one  excellent  maneuver  sector  is  read,  and  a least  squares  regression  is  performed  on  it.  The  data  are  read 
sector  by  sector  and  maneuver  by  maneuver  from  tape;  when  another  excellent  maneuver  sector  is 
encountered  it  is  used  to  update  the  regression. 

Four  candidate  reference  functions  are  generated  for  each  maneuver  sector: 

£ • 

* = 

^ = FAS  ^ ®Max*  ^Max) 
fyP^Max’^Max) 

®Max  » ^Max)  = Bo  + Bj  X + BjX?  + B30m  + B40m 

The  above  variables  are:  0m  ~ maximum  pitch  in  the  sector,  0m  = maxiumum  roll,  h = altitude,  0 = heading, 
AS  = airspeed,  0 = pitch,  and  the  "A”  means  “estimate”.  X,  the  independent  variable,  can  be  roll,  pitch  or 
normalized  time.  Roll  (0)  is  always  selected  first  if  roll  is  monotonic  in  that  sector.  If  roll  data  is  not 
monotonk  over  the  sector,  pitch  (0)  is  selected;  and  the  fourth  reference  equation  then  becomes: 

<t>  — Bo  + B|0  + Bjd5  + B3  0m  + B4  0m 


If  neither  pitch  nor  roll  is  monotonic,  normalized  time  (t)  is  selected  as  the  independent  variable,  and  in 
that  care  the  fourth  reference  function  can  have  either  0 or  0 as  the  dependent  variable. 

(3)  Performance  Discrimination 

The  purpose  of  a reference  function  is  to  specify  a standard  flight  path  for  use  in  developing 
performance  measures.  It  is  necessary,  therefore,  for  a refence  function  to  give  consistent  results  for  all 
excellent  performances,  and  at  the  same  time  provide  a basis  for  discriminating  performance  that  are  other 
than  excellent.  The  measure  used  for  a preliminary  test  is  the  mean  absolute  residual  error: 

T . 

e = 1/T  J IYj-Yj! 

1=1 


wherj^T  is  the  number  of  samples  in  the  maneuver  sector;  Yj  is  the  actual  value  of  the  dependent  variable; 
and  Yi  Is  the  prediction  of  Yj  by  the  reference  function.  The  value  of  e gives  an  indication  of  how  much  a 
sample  flight  deviates  from  the  reference  function.  If  the  reference  function  possesses  good  performance 
discrimination  capabilities,  then  e would  be  expected  to  be  small  for  excellent  Sights,  and  to  increase  as  the 
performance  level  worsens.  A graphical  interpretation  of  e appears  in  Figure  11. 

In  the  processor,  candidate  reference  functions  are  generated  from  several  excellent  maneuvers.  These 
maneuvers  can  be  considered  to  be  a “training  set”  for  the  processor  and  this  set  possesses  some  mean 
residual  error,  er,  with  respect  to  etch  reference  function.  As  a test  of  the  consistency  and  discrimination 
ability  of  the  reference  function,  ,ltest  sets”  are  formed  consisting  of  one  set  of  excellent  maneuvers  not 
included  in  the  training  set,  and  one  set  of  “poor”  maneuvers.  The  test  sets  also  have  a mean  residual  error. 
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Table  10.  Independent  Variable  Selected  for  Each  Maneuver  Sector 


Sector 


Maneuver  Typo 

i 

2 

3 

4 

s 

• 

7 

Cloverieaf 

Time 

Pitch 

Roll 

Pitch 

Pitch 

Splits 

Time 

Time 

Roll 

Roll 

Pitch 

Lazy  8 

Time 

Roll 

Roll 

Pitch 

Roll 

Normal  Landing 

Time 

Time 

Time 

Time 

Time 

Barrel  Roll 

Time 

Time 

Roll 

Ron 

RoU 

Ron 

Time 

Table  1 1.  Regression  Analysis  Results  for  Sector  1 of  the  C (overleaf 

Error* 


Exoolant  Poor 


Diptnwnx 

VftrittMt 

12SJL) 

i24(R) 

12  7<L) 

12S(ft) 

h 

25.44 

25.05 

25.73 

3794.50 

3478.60 

♦ 

0.49 

.09 

0.79 

.68 

2.79 

AS 

1.11 

1.14 

1.09 

3122.40 

2730.00 

0 

0.87 

.79 

.93 

17035 

146.99 

* 

0.34 

.36 

32 

1732 

15.76 

#/samples 

• 17 

23 

-7D 

-7.7 

5.4 

63 

-6.0 

-1.6 

-.4 

3.1 

FUNCTIONAL  FORM:  DV  = B0 

+ B,t'+Bjt2  + Bj 

0M  + ®4 

i 

Reference  Functions 

Voifabt* 

■o 

■l 

■a 

»3 

«« 

h 

2240.00 

-63120 

22D30 

318.10 

-141.80 

* 

-.40 

.44 

0.08 

-0.02 

-.29 

AS 

1890.00 

28.70 

-11.78 

254.90 

-175.70 

0 

90.58 

-23.55 

30.65 

14.05 

-8.48 

4 

10.22 

-2S1 

2.56 

137 

-0.09 

3s,,-w«wwv-«a»ff-w*-  - 


es,  with  respect  to  the  reference  function.  If  the  reference  function  is  a suitable  basis  for  performance 
discrimination,  then  es  should  be  approximately  equal  to  67  for  a test  set  comprising  excellent  maneuvers, 
and  significantly  greater  than  e?  for  a test  set  made  up  of  poor  maneuvers.  A ample  test  is  to  compare  the 
distribution  of  residuals,  e?,,  from  the  training  set  to  the  distribution  of  residuals,  e$i,  from  the  test  set  via 
the  rank  sum  statistic  (see  Rank  Sum  Statistic  section).  This  statistic  tests  the  hypothesis  that  the  two  sets 
(training  and  test)  are  equal;  i.e.,  that  the  reference  function  in  question  produces  similar  results  for  two 
different  sets  of  maneuvers.  The  rank  sum  test  is  used  because  it  requires  no  knowledge  about  the 
distribution  of  residuals,  and  it  can  compare  two  data  sets  of  different  lengths. 

c.  Results.  A total  of  19  maneuvers  were  run.  The  independent  variables  chosen  for  each  maneuver 
sector,  as  discussed  in  Regression  Analysis  section,  are  shown  in  Table  10.  Table  11  shows  the  actual 
reference  functions  for  sector  1 of  a cloverleaf.  The  functions  are  written  in  the  bottom  table  below  the 
functional  form.  The  residual  errors  (G)  appear  in  the  top  table.  Column  1 is  the  combined  error  of  the  two 
excellent  maneuvers  and  is  computed  as  follows: 

Nt  • Gt  ♦ N2  « g2 
c = N,  + N2 

where  N,  and  N2  are  the  number  of  samples  in  the  two  excellent  maneuvers  and  G,  and  Gj  are  their 
associated  errors.  Columns  4 and  5 show  the  errors  for  the  two  poor  maneuvers. 

Regression  functions  for  other  cloverleaf  sectors  and  other  maneuvers  appear  in  Appendix  A. 
(The  three  digit  numbers  for  the  maneuvers  were  assigned  for  maneuver  identification.  A summary  of 
maneuvers  used  is  shown  in  Table  12.)  Some  sectors  contained  too  few  data  points  to  obtain  a significant 
regression;  consequently,  no  data  appears  for  them. 


Table  12  Maneuver  Identification  Code  for 
Computer  Processor  Printouts 


M*n*uv*r 

Mod* 

Proficiency  Rating 

ID  Cod* 

Cloverleaf 

Right 

Excellent 

124 

Left 

Excellent 

125 

Left 

Poor 

127 

Right 

Poor 

128 

Split  S 

Left 

Excellent 

126 

Left 

Poor 

129 

Left 

Poor 

130 

Lazy  8 

Right 

Good  Plus 

102 

Right 

Poor 

106 

Left 

Ext  -. 

108 

Left 

Poor 

113 

Normal  Landing 

Excellent 

121 

Excellent 

122 

Poor 

123 

Poor 

124 

Barrel  Roll 

Left 

F“^Hent 

101 

Left 

•ilent 

107 

Left 

Poor 

no 

Left 

Poor 

114 

On  certain  maneuvers,  very  small  differences  in  $Max  and  seem  to  cause  large  differences  in  the 
residual  errors.'  For  example,  in  sector  6 of  the  normal  landings  (Appendix  A),  the  coefficient  of  dMax  for 
the  heading  regression  is  extremely  large  (ri4  * -7990).  Although  of  maneuver  124  differs  from  <$m  of 
the  excellents  by  only  about  two  degrees,  the  difference  in  €*g  is  huge.  Based  on  the  processing  of  four 
maneuvers,  it  would  appear  that  an  excellent  maneuver  with  a slightly  different  could  significantly 
increase  the  €’s  for  the  excellent  maneuvers. 


Further  processing  is  required  to  determine  if  this  is  true  in  general.  If  sc,  one  alternative  would  be  to 
change  the  form  of  the  regression;  e.g.,Y  = B0+BiX+B2XJ.  However,  lacking  the  necessary  data  to  test 
many  more  maneuvers,  as  required,  the  regression  functions  in  the  processor  were  implemented  as 
documented  in  this  report 


Adaptive  Math  Models 

This  section  describes  the  experimental  techniques  for  generating  candidate  measures  for  subsequent 
validation-testing.  The  models  for  so  doing  are  called  adaptive  mathematical  models  because  the  candidate 
measures  which  they  generate  are  derived  recursively  and  adaptively  in  accordance  with  the  success 
encountered  with  various  measure-types.  Since  much  of  the  underlying  mathematics  has  been  documented 
in  earlier  referenced  studies,  emphasis  here  will  be  on  a brief  description  of  each  model  and  those  areas 
where  refinements  were  incorporated  as  a part  of  this  study. 

The  purpose  of  the  adaptive  math  models  (AMM)  is  to  systematically  search  Boolean  time  sequences 
(BTS)  for  various  characteristics  and  determine  if  the  characteristics  are  related  to  performance 
measurement.  A block  diagram  of  this  process  is  shown  in  Figure  12.  Smoothed  flight  data  is  directed  to 
Boolean  logic  which  processes  the  data  and  develops  Boolean  functions  designed  to  succinctly  represent 
critical  performance-reWed  information  contained  in  the  data.  The  output  from  the  Boolean  logic  is  a set 
of  Boolean  time  sequen  ;es  which  are  directed  to  three  processes:  relative,  absolute,  and  state  transfer.  Each 
of  these  processes  searches  for  different  types  of  measures,  as  discussed  in  the  Data  Smoothing  section.  The 
processes  are  two-step  operations  in  which  first,  the  Boolean  time  sequences  are  systematically  searched  for 
characteristics  and  relationships  potentially  related  to  performance  measurement.  When  useful 
characteristics  are  detected,  a test  is  conducted  to  determine  their  significance  to  measurement.  Finally,  the 
outputs  of  each  of  the  three  processes,  which  are  intermediate  performance  measures,  are  combined  in  a 
weighted  sum  to  provide  an  overall  performance  measure  for  evaluation. 


To  establL*  the  notation  used  in  following  sections,  consider  a Boolean  time  sequence  where  a single 
bit  of  the  seqi  .nee  is  represented  by  BTSy.  The  first  subscript  (i)  represents  the  Boolean  function  which 
generates  the  BTS,  and  the  second  subscript  0)  identifies  the  jth  element  of  that  sequence.  Thus,  Boolean 
time  sequence  i is  given  by: 


BTSs;j  = l,Mk 


where  Mj,  is  the  number  of  elements  in  the  sequence.  The  superscript  (k)  is  used  to  indicate  the  flight 
event  or  flight  maneuver  number  associated  with  the  Boolean  sequence.  It  b seen  that  Mfc  is  a function  of  k 
only  and  not  i,  because  every  Boolean  sequence  generated  with  data  from  flight  event  k has  the  same 
length.  When  reference  is  made  to  a total  BTS  for  a specified  flight  event,  the  notation  BTS*  is  used. 


a.  Bookjx  Function  Data  Representation.  A special  transformation  of  the  data  is  performed  to 
simplify  its  analysis  and  to  permit  the  user  to  interact  with  the  processor  by  adding  to  it  his  knowledge  of 
the  problem.  The  transformation  results  in  representation  of  the  data  in  the  form  of  BTS  produced  by 
applying  a sequence  of  performance  demonstration  data  samples  to  Boolean  functions  (BF).  Two  types  of 
BF  are  constructed  and  will  be  discussed  in  turn: 
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Figure  11  Block  diagram  of  adaptive  nuth  modd  «e*rch  pcoceat. 


• ' ' Functions  representing  the  raw  data  itself,  *u  expressed  in'tenm  of  discretized  deviations  from 
criterion  (of  reference)  functions;  and 

9k,  Functions  which  are  demanded  by  the  user  and  reflect  his  knowledge  about  features  of  the 
performance  that  are  considered  by  him  to  be  relevant  to  measurement. 

(1)  Functions  Representing  the  Raw  Data 

Without  relying  on  guidance  from  the  user,  BF  are  constructed  by  modeling  the  deviation  of  the 
various  parameters  from  standard  profiles  or  time-histories.  This  is  accomplished  by,  fust,  spanning  the 
envelope  of  performance  with  amplitude  test  bands,  as  filustiated  in  Figure  13.  The  test  bands  (labeled  1 
through  5 in  Figure  13)  are  determined  empirically  and  consist  of  multiples  of  standard  deviations  around 
the  reference  function,  computed  uring  performances  of  skilled  subjects.  BF  are  constructed  to  represent 
activity  in  each  test  band,  the  BF  are  set  true  only  when  the  actual  performance  data  sample  is  within  the 
limits  of  the  respective  test  bands. 

(2)  Functions  Demanded  by  the  User 

The  user  may  construct  special  BF  by  asking  pertinent  yetor-no  questions  about  the  performance. 
The  answers  (1  or  p)  then  form  fire  values  of  the  associated  BTS.  For  instance,  the  user  may  have  reason  to 
believe  that  whether  or  not  a pilot’s  turns  are  consistently  coordinated  Is  particularly  relevant  to  measuring 
his  performance  on  a given  maneuver.  Therefore,  he  may  ask  whether 


Roll  = f(pitch,  rate-of-tum,  . . .)  ± 6 

where  the  function  f is  designed  to  model  a coordinated  turn.  If  indeed  this  information  is  relevant  to 
measurement,  then  the  level  of  activity  (percentage  of  time  true)  of  the  associated  BTS  would  probably  be 
a good  performance  measure. 


As  a second  example,  there  may  be  reason  to  believe  that  performance  at  critical  points  in  the 
maneuver  is  particularly  relevant.  To  augment  the  processor  with  this  information,  the  user  might  pose  the 
question,  “is  the  performance  currently  at  a critical  point  (like  Pitch  * Max  Pitch  ± 10°)?”  The  resulting  BF 
then  identifies  that  paint  in  the  maneuver.  In  this  particular  case,  the  associated  BIS  itself  tasty  not  be 
relevant;  rather,  its  logical  relationship  with  other  BTS  (Le-.  what  is  happening  at  the  critical  point)  would 
probably  be  of  most  value. 

b.  Absolute  Measures.  The  absolute  computation  mechanism  consists  of  a correlation  of  each  BTS 
against  a fixed  set  of  functions  or  sequences  (MacDonald  Codes).  This  results  in  the  transformation  of  a 
long  sequence  (BTS)  into  a new  set  of  non-Booiean  variables  which  in  turn  can  be  examined  to  determine  if 
they  are  relevant  to  performance  evaluation. 

Correlation  against  an  absolute  reference  allows  a search  for  measurement-significance  of  particular 
sequences  or  patterns  as' they  are  generated  by  the  Boolean  functions.  If  it  is  found  that  some  BTS  pattern 
is  likely  to  be  predictive  of  skilled  operator  performance,  this  information  can  serve  as  a basis  for 
specification  of  automatic  scoring  systems  as  well  as- provide  clues  about  the  operator  techniques  used  in 
achieving  superior  performance.  The  absolute  measure  also  allows  analysis  employing  multiple  BTS  as  well 
at  a single  BTS  via  a regression  computation.  This  provides  the  tools  required  for  a systematic  study  of 
which  Boolean  function  and  combinations  thereof  are  relevant  to  measurement. 

The  absolute  computation  is  defined  as: 


aL_ 

« Nj  -N, 


N, 

I 

j*N, 


®TS}xZQ,NJ-NI<Mk 


where  Zq  is  an  element  in  a reference  sequence.  The  subscript  (f)  indicates  which  reference  sequence  is 
being  used.  Note  that  the  summation  is  riot  conducted  over  the  toUi  length  of  the  BTS;  rather,  it  is 
computed  over  a short  interval  of  the  BIS.  There  are  two  factors  that  lead  to  this  approach.  First,  every 
performance  does  not  require  the  same  length  of  time  and  as  a result  M*  is  not  a constant.  Thus,  the 


summation  must  not  be  over  an  interval  greater  than  the  smallest  value  of  M*..  Second,  and  far  more 
important,  is  that  the  set  of  reverences  f]  should  be  a set  of  orthogonal  binary  sequences  in  order  to  have  an 
efficient  reference  set.  (It  can  be  shown  that  there  arc  N such  sequences  N elements  long;  i.e.,  fj  is  a square 
matrix.)  The  absolute  measure  can  also  operate  on  threshold  variables  (see  Adaptive  Mathametrical  Models 
section)  instead  of  the  BTS.  In  that  case,  the  computation  is  defined  as: 

N2 

c£'N,  -N,  2 RjxZ0,N,-N,  <Mk 
j=N, 

The  process  implemented  in  the  computer  program  uses  a Hadamard  transform  producing  coefficients 
from  all  reference  sequences  in  one  efficient  operation  rather  than  N operations.  However,  the  process  is 
described  by  the  equivalent  correlation  operation  with  a reference  sequence. 

The  length  of  the  reference  sequence  (and  therefore  the  number  of  reference  sequences)  is  taken  as  a 
varaible  2M,  M - 2,  3,  4,  5,  6,  in  order  to  facilitate  generation  of  the  Hadamard  transform.  As  stated 
previously,  the  reference  sequence  length  must  be  adjusted  to  accommodate  the  length  of  the  BTS; 
however,  the  length  of  the  BTS  is  not  the  only  factor  of  interest.  The  optimum  length  of  the  reference 
sequence  required  to  produce  sensitive  performance  measures  is  not  known.  It  is  known  that  the  BTS 
pattern  can  be  searched  in  several  ways.  Two  possible  ways  are  shown  in  Figure  14.  Method  A shown  in  the 
figure  requires  a correlation  of  the  4 bit  reference  sequence  to  4 bits  (in  general  2M)  of  the  BTS  followed 
by  correlation  of  the  reference  sequence  with  BTS  bits  5-8,  etc.,  until  all  BTS  bits  have  been  processed. 
Since  there  are  4 reference  sequences  (of  4 bits  each)  the  process  is  a multi-pass  operation.  The  equivalent 
transform  operation  requires  one  pass.  It  can  be  shown  that  the  values  of  the  4 coefficients  for  each  shift 
uniquely  specify  the  BTS  and  no  information  is  lost  by  the  correlation  (transform)  operation.  (Preservation 
of  information  may  or  may  not  be  necessary  or  a sufficient  requirement  in  performance  measurement.  In 
fact,  it  is  easily  seen  that  performance  measurement  is  an  operation  in  which  information  is  discarded 
systematically,  thus  reducing  a great  volume  of  data  to  a few  variable  values  representing  performance.) 

Method  B employs  a correlation  (transform)  operation  followed  by  a shift  of  one  bit,  followed  by  a 
second  correlation,  etc.  This  method  allows  examination  of  each  sequence  of  2M  bits  and  for  that  reason  is 
preferred.  Various  length  reference  sequences  can  be  processed  without  risk  of  an  incomplete  correlation  at 
the  end,  due  to  a BTS  length  not  equal  to  a multiple  of  2M.  Thus,  the  processor  is  designed  to  employ  refer- 
ence lengths  2M,  M = 2, 3, 4,  5, 6,  and  use  the  shift  pattern  shown  as  Method  B in  Figure  14. 

Each  correlation  operation  produces  one  value  of  the  correlation  coefficient  and  there  arc  N + 1 
correlations,  where  N is  the  number  of  shifts.  If  the  BTS  has  L bits  and  the  length  of  the  reference  sequence 
is  2M,  there  are  L— 2M  + 1 correlation  coefficients  (C)  for  each  BTS  and  reference  sequence  combination. 

Detection  of  “patterns”  in  each  BTS  is  accomplished  by  analysis  of  the  distribution  of  the  correlation 
coefficient  (C)  values  obtained  from  each  channel  (BTS  and  reference  sequence  combination).  A 
fundamental  question  is  determining  the  C distribution  that  might  result  from  a random  BTS  (i.e.,  without 
consistent  patterns).  Consider  a random  BTS  where  each  bit  of  the  sequence  has  a probability  of  0.5  of 
being  a 1 or  - I.  This  population  has  a mean  and  variance  of: 

H = 0 

°2~  (-1)2  x .5  + 1*  x .5  = 1 

The  correlation  operation  can  be  considered  as  a summation  of  N elements  or  that  population  and  the 
distribution  of  summation  has  a mean  and  variance 

Mn  = N/i  =0 

oN2  = No*  *N 


Figure  14.  Methods  of  pattern  search. 


The  distribution  of  on3  is  of  specific  interest  since  a computed  sample  variance  from  flight  data  is  to  be 
used  to  detect  the  existence  of  data  patterns.  Thus,  the  90  and  95  probability  points  of  the  cumulative 
distribution  for  on  2 are  denied  for  an  automatic  decision  threshold  so  that  the  number  of  false  pattern 
detections  (Type  I error)  can  be  controlled.  Analysis  of  this  problem  resulted  in  the  identification  of 
appropriate  decision  thresholds  for  incorporation  in  the  processor. 

c.  Method  of  Constructing  System  States  for  State  Transfer  Measures,  ftevious  efforts  in 
constructing  Boolean  functions  for  the  purpose  of  proficiency  measurement,  employed  one  or  more 
threshold  states  for  each  flight  variable  (Connelly  et  al.,  1969  & 1971).  ft  was  observed  in  those  studies  that 
improved  results  could  be  obtained  using  a new  variable  derived  from  deviations  from  reference  functions. 
For  example,  in  the  lazy  8 maneuver,  a function  relating  pitch  and  roll  angles  was  used  to  provide  a 
reference.  Threshold  states  were  constructed  as  Ko  displacements  (i-e.,  multiples  of  standard  deviations) 
from  this  reference  function.  These  threshold  states,  sequences  of  threshold  states,  and  patterns  of 
threshold  states  are  used  to  obtain  estimates  of  the  system  performance. 

While  this  method  works  well,  it  is  desirable  to  extend  the  method  such  that  threshold  states  for  more 
than  one  reference  function  can  be  considered  collectively.  For  example,  a state  transition  measure  should 
be  more  effective  where  the  states  reflect  collective  deviations  from  more  than  one  reference.  However,  the 
number  of  combinations  of  threshold  states  can  be  large  which  leads  to  problems  in  computation,  data 
storage,  and  data  collection. 

It  is  possible  to  combine  several  threshold  state  combinations  into  performance  states  with  what  is 
believed  to  be  a reasonable  way.  Benefits  from  such  an  approach  include  simplified  computation  while 
maintaining  a “physical”  interpretation  of  the  performance  states.  It  is  necessary  to  compare  each  sector  of 
actual  data  to  this  reference  path  to  obtain  an  indication  of  how  much  deviation  exists  between  the  two 
(Figure  15).  Instead  of  listing  the  sequence  of  the  reriduals  (e*  ~ 0$  -$})  for  each  sample,  a new  sequence 
can  be  written  as  follows.  If  |ej|  is  less  than  la,  define  R#  = 0.  If  ej  is  between  la  and  2a,  let  R#  = 1,  etc. 
Now,  the  sequence  of  residuals  is  reduced  to  a sequence  which  contains  values  0-4  which  are  states.  The 
sequence  Rg  can  be  considered  as  a function  state  since  it  shows  the  progression  of  states  followed  by  the 
flight  This  process  can  be  generalized  as  follows: 

Consider  a set  of  reference  functions  as  follows: 

= fg(*,0Max,0Max) 
fAS  (p,  0Max,  dMax) 
ir'=  fh  ($, 0Max,  4>Max) 

^ * f ($,  0Max,  $Max) 
where 

t it  roll  angle 
0 is  pitch  angle 
AS  is  airspeed 
h n altitude 
<1*  is  heading 

A * 

x is  the  estimate  of  variable  x. 

• The  error  function  for  0 is  given  by: 

% =0-f9*Ed(0,0,dMax,0Max) 

^a*  %e<MMax,0Max) 
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Sequence  of  residuals;  j Cj  e2,  63,  e4 „T  I 

Sequence  R : I 1,  1,  1,  2,  2,  3,  3,  4,  A,  3,  1,  ... 


Boolean  Time  Sequence: 


BTS  = 


(.0  R < 2 

'1  R > 2 


- { 0,  0,  0,  0,  0.  1,  1,  1,  1,  t,  0 

I 

1 Figureis.  Devktioa  betw*«a  acted  data  and  reference  pith. 


where  the  value  of  Eqo  is onc  sigma  deviation  function  for  the  distribution  of  0 given  (0,  0Max,  0Max). 
In  a similar  way,  additional  error  and  onc  sigma  deviation  functions  can  be  formed 
Eas  (AS,  0, 0Max,  0Max) 

EASo  0Max>  0Max) 

(h,  0,  0Max,  0Max) 

E-na  (0,  OMax,  0Max) 

E^  (0, 0, 0Mux,  0Max)  . 

E^a  (0,  0Max,  0Max) 

Define  threshold  variables  as: 

Rx  = 0 when  | Ex  | < k EXa 

Rx  = 1 when  KEXa  < |EX|  < 2k  EX(J 
Rx  = 2 when  2kEXo  < (E^  < 3k  EX(J 
Rx=3  when  3kEXff  < |EXI<  4k  EXo 

Rx=4  when  4kEXo  < |EXI 
where  X =0,  AS,  h,  0. 

A system  state  can  be  formulated  as: 

S = R0  + RAS  + Rfc  + R0 


Thus,  S values  range  from  0 to  16.  The  following  diagram  illustrates  the  translation  between  threshold 
regions  activity  and  the  system  state  representation. 

S R Ras  Rk  R 


0 0 0 0 0 
110  0 0 
10  10  0 
10  0 10 
1 0 0 0 1 
2 110  0 
2 0 110 
2 0 0 11 
2 2 0 0 0 


16 


4 


4 


4 4 


This  method  provides  a summary  performance  state  which  renders  several  individual  threshold  states 
equivalent.  The  method  allows  a compact  17  state  representation  of  a system  that  contains  many  more 
states. 

In  summary,  there  are  several  types  of  system  states  available.  These  arc: 

• Binary  Threshold  states 

A binary  threshold  state  is  defined  by  a binary  valued  function  indicating  if  the  present  BTS 
sample  (residual)  exceeds  a specified  level,  i.c., 
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BTS  * 1 if  R.j  > k 

«j  ij  i 

^O.fR^k, 

where  is  the  threshold  variable  and  k is  the  associated  threshold  factor. 

• Threshold  slates 

A threshold  state  is  the  value  of  the  threshold  variable  Rx  = 0,  1 , 2, . 

• System  states 

A irerformancc  state  is  the  value  of  the  system  vartablc(s)  given  by  the  sum  of  the  threshold 

variables  S = Rq  + RAS  + R(|  + R^ 

The  BTS  is  a sequence  with  I/O  valued  elements.  In  a strict  sense,  only  binary  threshold  states  can  be 
represented  in  a BTS;  however,  for  computation  of  transition  measures  threshold  and  system  variables  are 
useful.  In  the  state  transfer  computational  mechanism,  the  multi-valued  variables  R and  S are  used  to 
replace  the  binary  valued  BTS. 

d.  State  Transfer  Measures.  The  state  transfer  computation  mechanism  is  a means  for  determining  if 
performance  (or  score)  information  is  related  to  the  sequence  of  operator  actions.  It  may  be  that  operator’s 
performance  is  partially  or  totally  a function  of  how  he  corrects  for  errors,  where  he  may  or  may  not  have 
caused  the  errors  initially. 

In  order  to  implement  this  computation  and  also  provide  a convenient  means  for  compactly 
representing  long  Boolean  time  sequences,  a transition  matrix  is  formed  which  identifies  how  the  sequence 
moves  from  state  to  state.  Also,  a composite  transition  matrix  can  be  formed  to  represent  transition 
patterns  from  all  demonstration  data  sets  (DDS)  of  a given  performance  level. 

A Boolean  state  is  defined  by  the  set  of  binary  values  associated  with  the  set  of  selected  BTS  as 
described  in  the  previous  section.  Each  DDS  can  be  viewed  as  a sequence  of  Boolean  states  or,  alternatively, 
as  a sequence  of  state  transitions  termed  "translates.”  The  state  transition  measure  seeks  to  relate  the 
frequency  of  use  of  each  transtate  to  performance  measurement.  This  is  accomplished  by  associating  with 
each  transtate  a score  value  stored  in  an  incremental  score  matrix  (ISM).  A performance  measure  value  is 
produced  by  summing  the  score  values  from  the  element  of  the  ISM  corresponding  to  each  translate  used  In 
the  DDS.  The  final  performance  measure  value  is  obtained  by  dividing  the  sum  by  the  number  of 
transitions.  The  transition  matrix  is  formed  using  the  system  state  values 

S = 0,  1,2 16 

A 17  x 17  transition  matrix  is  required  to  store  all  transition  probabilities. 

If  we  assume  that  the  DDS  state  sequence  can  be  described  as  a Markov  Process,  the  sequence  can  be 
represented  by  its  transition  matrix.  Under  this  assumption,  the  performance  measure  can  be  computed  in 
another  way.  A fundamental  theorem  for  Markov  Chain  processes  states  that  if  trD  is  the  initial  probability 
vector  (probability  distribution  density),  then  (the  probability  vector  after  n trials)  Is  given  by 


where  T is  the  process  transition  matrix.  Proof  of  the  results  given  herfc  can  be  found  in  Connelly  et  a!.. 
1969. 

Now,  we  assume  that  the  process  b a regular  Markov  process.  Such  a process  Is  identified  by  a 
transition  matrix  (T)  where  for  some  value  of  n.  Tn  has  no  zero  elements.  Thb  Implies  that  th*  system 
could  be  in  any  state  after  N trials,  independent  of  the  initial  state. 


.1? 


The  assumption  that  the  system  is  a regular  Markov  process  lets  us  state  that  there  exists  a unique 
probability  vector  (a)  such  that 

as  n -*■«»,  limit  v =a  (2) 

Elements  of  this  probability  vector  correspond  to  the  probability  of  the  system  being  in  the  associated 
state. 

Furthermore,  it  is  seen  that 


aT  = a (3) 

must  hold  due  to  Equations  (1)  and  (2).  This  limiting  probability  vector  gives  the  first  state  distribution 
desired.  Note  that  a is  the  probability  distribution  of  finding  the  process  in  each  state,  given  that  we  have 
not  observed  the  process  previously. 

The  distribution  of  translates  can  be  determined  by  imagining  an  ensemble  of  many  adjustment 
systems  with  first  states  distributed  according  to  a.  The  second  states  for  each  system  are  determined 
according  to  the  transition  matrix  T.  This  yields  new  states,  also  with  a distribution  a (according  to 
Equation  3).  The  probability  that  transition  i-j  is  used  in  the  operation  is  the  probability  of  the  transition 
(Ty)  given  the  first  state  times  the  probability  of  being  in  state  i (aj). 

Let  a be  an  N by  N matrix  with  zero  value  elements  off  the  main  diagonal.  Also,  the  elements  (a'jj) 
of  a are  given  by 


A new  NxN  matrix (D) is  defined  as 


D = a'T 


Elements  of  D (oj  Ty)  are  the  probabilities  of  the  system  being  in  each  transtate  assuming  the  first  (or  any 
other)  state  is  not  known. 

The  probability  matrix  (D)  can  be  used  to  establish  the  equivalent  population  statistics.  Elements  of 
D (ij)  may  be  considered  normalized  weighting  factors,  and  elements  of  ISM  (ij)  provide  the  population 
values.  The  population  mean  (P)  is 

N N 

P=  2 I D (i  j)  ISM(i  j) 
i=l  j=l 

P is  equivalent  to  the  performance  measure  computed  from  state  transition  as  described  previously.  In 
addition,  we  now  have  the  tools  for  computing  values  for  the  ISM.  The  method  is  to  form  a representative 
transition  matrix  for  two  or  more  performance  levels  of  the  DDS.  In  this  way,  one  transition  matrix 
represents  excellent  performance  and  another  represents  another  performance  level,  etc.  Once  these 
composite  transition  matrices  are  available,  the  elements  of  ISM  can  be  adjusted  (trained)  to  improve  the 
• performance  measure  discrimination  capability  between  (or  among)  the  demonstrated  performance 
categories.  The  method  is  to  sequentially  adjust  the  ISM  using  one  transition  matrix  at  a time  and  to 
continue  the  process  until  a measure  with  stable  discrimination  capability  is  obtained.  There  are  alternative 
methods  of  adjusting  the  ISM,  but  this  iterative  method  converges  rapidly  and  allows  introduction  of  new 
data  as  it  is  obtained.  The  iterative  method  requires  computation  of  the  amount  each  element  in  ISM 
should  be  changed  in  order  to  modify  the  score  (measure  value)  by  a specified  amount. 

The  probability  matrix  D (ij)  can  be  used  to  compute  the  expected  value  of  score  change  by  means 
of  the  adjustment  process.  For  each  transition,  the  probability  that  transtate  (ij)  is  used  (assuming  wc  do 
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not  know  the  present  state)  is  D (ij).  Thus,  the  expected  change  in  the  associated  incremental  score,  ISM 
(ij),  at  each  transition  is  the  product 

A-DGj)  . 

where  A is  the  amount  added  to  the  ISM  increment.  Since  n2  transitions  are  used  in  the  sequence,  the  total 
expected  change  is  given  by: 


n2  • A • D(i  j) 

If  we  use  ISM  (i  j)'  to  represent  the  expected  updated  incremental  score  matrix,  we  see  that: 

ISM  (ij)'  = ISM  (ij)  + n2  • A • D(ij) 

The  expected  mean  score  is 

I"=  2 2 D(ij)  * ISM(ij)  + n2  * A • (D(i  j))1 

i i 

. and  the  expected  change  in  mean  score  (CM)  is 

CM  - n2  • A • Z ZOXij))1 
i j 


The  summation  term  can  be  considered  as  the  system  gain  G,  such  that 

g=  now 

i j 

(Note  that  G will  be  less  than  one.) 

Thus,  the  amount  that  must  be  added  to  each  element  of  ISM  is  n2  • A • D(i  j)  in  order  to  change  the 
measure  value  by  amount  CM. 

e.  Relative  Measures.  The  relative  computation  technique  operates  on  up  to  four  Boolean  functions 
simultaneously  to  determine  if  logical  relationships  exist  and,  if  so,  how  the  relationships  are  associated 
with  performance.  Thus,  as  opposed  to  the  absolute  computation  where  operations  are  performed  on  a 
single  BTS,  the  relative  computational  technique  uses  a “trainable  logic”  concept  to  detect  possible 
relatioruhips  among  BTS.  The  approach  is  to  select  a set  of  base  BF  channels  and  form  all  combinations  of 
these  sequences  for  each  data  sample.  For  example,  if  three  base  BF  channels  are  selected,  eight 
combinations  can  be  produced.  Only  one  combination  is  true  for  each  sample.  Correlation  of  one 
combination  (say  combination  j)  with  an  additional  BF  channel  (say  BFj)  yields  the  conditional  probability 
that  BF{  is  true  given  that  combination  j is  true. 

Bach  combination  can  be  expressed  as : 


where  the  tint  subscript  indicates  the  BTS,  and  the  superscript  indicates  the  DOS.  The  second  subscript  0) 
indicates  the  element  In  the  sequence.  The  AND  operation  is  conducted  on  a bit’by-bit  basis.  Therefore,  the 
combinat]qn  Cm(j)  has  a binary  value  corresponding  to  each  element  in  the  BTS;  i.e„  Cm(j)  is  a Boolean 
sequence  itself. 

■As  an  exan^le,  we  determine  combination  S.  It  is  convenient  to  represent  m (or  5 in  this  case)  by  its 
binary  form  (101)7  Thus,  uting  the  binary  form  of  m to  code  the  combination,  we  find 
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Cj(j)  - BTS^j  BTsJ  BTS^ 


where  BTS  is  the  NOT  function.  Note  that  the  second  subscript  need  not  be  the  same  for  each  term;  i.e,  we 
can  compute  using  relative  time  positioning  by  selecting  (for  example)  second  subscripts  (j,j+2,  j+3,j-l). 
While  this  flexibility  is  available  it  has  not  been  extensively  explored  to  date. 

The  analysis  consists  of  the  following  operations.  First,  a number,  SUMn,  is  computed  which  is  a 
count  of  the  number  of  times  each  combination  occurs  during  the  flight  event  under  study.  Titus, 


sumn  = 


Mk 

2 

j=i 


C„G)- 


A second  count  is  formed  which  represents  the  number  of  times  a BTS  variable  is  true,  given  that 
combination  Cn  is  true.  Thus, 


SUMC  = 


Mk 

2 

j=i 


'„0>BTScj 


Normally,  the  first  subscript  e would  have  a different  value  than  those  of  the  base  variables.  Next,  the 
conditional  probability  that  BTS^j  is  true  given  that  combination  n has  occurred  is  determined  as 


P>)  = 


SUMC  » 


SUM, 


N 


This  conditional  probability  identifies  the  relationship  between  each  combination  of  the  base  variables  and 
the  predicted  variable.  These  conditional  probabilities  are  candidate  performance  measurement  variables 
and  are  tested  for  validity,  as  described  in  Validation  Tests  section. 

f.  Summary  Description  of  Measures.  Figure  16  illustrates  a 2-dimensional  state-space  defined  by  roll 
and  pitch  angles,  and  the  approximate  trajectory  outline  of  one  quarter  of  a lazy  8 maneuver.  The  small 
arrows  in  the  figure  depict  alternative  directions  in  which  a roll/pitch  trajectory  might  move  in  a given 
performance.  The  State  Transfer  type  of  measure  is  based  on  probabilistic  assessments  of  this  direction  of 
movement.  To  compute  the  probability  values,  the  state-space  is  gridded  into  discrete  states.  (For  instance, 
each  cell  or  rectangle  in  Figure  16  may  be  considered  an  individual  state.)  By  computing  the  frequency  with 
which  each  state  is  acquired  and  the  state  sequence,  the  probability  of  transfer  from  state  to  state  is 
calculated. 

The  State  Transfer  computational  mechanism  can  operate  on  up  to  four  flight  variables  at  a time.  To 
minimize  the  number  of  states  to  be  handled  simultaneously  and  the  associated  computational  complexity 
of  the  problem,  threshold  states  are  used  which  represent  the  sum  of  the  deviation  units  from  each  criterion 
function.  For  example,  in  Figure  16  the  shaded  cells  might  be  recorded  as  threshold  state  number  1 
(depending  on  computed  performance  variance  and  resulting  cell  sizes  used  to  model  each  performance), 
because  they  are  located  one  deviation  unit  from  the  reference  trajectory.  This  state  representation  not 
only  reduces  problem  complexity,  but  permits  ready  interpretation  and  assessment  of  divergence  from  or 
convergence  on  criterion  terminal  performance. 

The  Relative  measure  is  based  on  conditional  probabilities  of  various  states  being  acquired 
simultaneously  with  the  acquisition  of  other  states.  For  instance,  consider  the  user-defined  BTFs  of  (1) 
Pitch  * maximum  pitch  *Aj , and  (2)  airspeed  = Ar  ± Aj , where  Ar  is  a criterion  airspeed  value.  Analyzing 


these  BTFs  along  with  the  information  in  Figure  16,  the  Relative  computational  mechanism  would 
examine  as  a potential  measure  the  probability  that  pitch  is  maximum  whenever  airspeed  = A*  and  state  X 
in  Figure  16  is  active.  (This  is  precisely  a significant  factor  of  criterion  lazy  8 performance  as  suggested  by 
Air  Training  Command  flight  manuals  and  as  substantiated  empirically  in  previous  studies  (Knoop  & Welde, 
1973)).  This  is  only  one  example  of  hundreds  where  simultaneity  of  significant  events  bears  on  successful 
performance,  and  the  role  of  the  Relative  computational  mechanism  is  to  explore  the  relevance  to 
performance  measurement  of  conditional  probabilities.  When  one  considers  the  plausible  theory  that  much 
of  performance  on  continuous  control  tasks  can  be  modeled  by  discrete  successive  acquisitions  of  key 
states,  the  value  of  the  Relative  type  of  measure  and  its  role  in  the  processor  becomes  clear. 

The  Absolute  computational  mechanism  essentially  amounts  to  computing  a discrete  verson 
(Hadamard  Transform)  of  the  Fourier  Transform,  wherein  the  power  of  various  frequency  components  of 


51 


S-- 


the  signal  (BTS)  is  assessed.  Each  transform  operation  produces  a number  of  coefficients  (correlation 
values),  which  are  summarized  by  their  mean  and  variance.  Previous  work  (Connelly,  et.  al.,  1969)  has 
shown  that  the  variance  of  the  Hcdamard  coefficients  is  most  useful  in  discriminating  performances  of 
various  skill  levels.  The  Absolute  mechanism  generates  these  variances  for  subsequent  validation  testing, 
described  next. 


Validation  Tests 


a.  Description.  For  the  vast  majority  of  performance  tasks,  there  is  no  single  necessary  and  sufficient 
test  that  can  be  applied  to  candidate  measures  to  assess  their  validity.  Measures  which  appear  to  have 
content  validity  often  fail  to  reliably  discriminate  even  between  novice  and  highly  experienced  performers. 
Measures  which  appear  to  have  concurrent  validity  may  or  may  not  satisfy  other  validation  criteria, 
depending  on  the  reliability  and  sensitivity  of  the  metric  used  as  a baas  of  comparison. 

The  approach  in  this  study  was  to  develop  three  empirically-based  validation  tests  to  be  applied  by 
the  measurement  processor.  Collectively,  the  tests  are  used  to  determine  the  likelihood  that  each  candidate 
measure  is  valid.  Final  analysis  and  assurance  of  the  measure’s  content  validity  is  performed  by  the  user  of 
the  processor,  based  on  the  evidence  accrued  by  it  and  printed  out  for  his  consideration. 

The  first  test  assesses  the  measure's  potential  contribution  to  discriminating  between  performances  at 
opposite  ends  of  the  skill  continuum.  The  data  employed  for  this  test  arc  selected  by  the  user.  For  the 
T-37  pilot  performance  tasks  that  were  to  have  been  addressed  here,  the  following  two  types  of  data  would 
have  been  investigated: 

(1)  Flights  flown  by  instructor  pilots  to  demonstrate  their  best  performances  and  simulated  novice 
performances  of  each  maneuver. 


(2)  Flights  flown  by  students  at  the  neophyte  stage  and  at  the  successful  completion  of  training. 

The  techniques  implemented  to  apply  this  first  test  include:  (a)  comparison  of  residues  from  regression 
analyses,  and  (b)  the  rank  sum  statistic  (see  Validation  Tests  section). 

The  second  test  assesses  the  measure’s  functional  relationships  with  variables  such  as  number  o(  trials 
and  time  in  training.  A measure  which  demonstrates  that  learning  has  occurred  from  neophyte  to 
experienced  levels  of  performance  would  posses  a higher  likelihood  of  validity  than  one  which  consistently 
does  not,  for  example.  Again,  the  data  to  be  employed  for  this  test  are  specifiable  by  the  user.  For  the  T-37 
pilot  tasks,  the  following  data  would  have  been  experimented  with:  (1)  time  in  training,  (2)  Number  of 
practice  sorties  on  the  maneuver,  and  (3)  number  of  practice  trials  on  the  maneuver.  The  technique  used  to 
apply  this  test  consists  of  developing  and  analyzing  a multi-variable  regression  function.  (An  alternative 
technique  based  on  the  use  of  Markov  learning  models  was  conceived,  but  due  to  lack  of  data,  has  not  yet 
been  developed  to  the  point  of  implementation.) 

The  third  test  assesses  the  measure’s  functional  relationships  with  subjectively  derived  ordinal  scale 
measures  of  performance.  Measures  which  tend  to  reinforce  the  subjective  ordering  of  performances  are 
considered  more  likely  to  be  valid  than  those  which  consistently  fail  to  do  so.  The  data  employed  for  this 
test,  ^ min  die  other  tests,  are  specified  by  the  user.  For  the  T-37  tasks,  instructor  pilot  ratings  would  have 
been  investigated  for  use.  The  technique  for  app.,ing  the  test  is  to  develop  and  analyze  multi-variable 
regression  functions,  as  in  the  second  test  described  in  the  preceding  paragraph. 

r5rCSS,C-‘  tcchRiqu«  used  for  applying  some  of  the  above  validation  tests  were  described 
previously.  The  rank  sum  test  is  described  next. 

„ S,T  Th'  computer-aided  generation  of  performance  measures  requires  the 

S«C  ^ evaluation  of  many  candidate  measures,  ft  is  necessary  to  assess  'hese  measures* 

contribution  to  overall  performance  measurement.  One  aid  in  accomplishing  this  using  the  rank 
sum  statistic  was  developed  for  investigation  in  this  study.  ™ 

• , Ctonsid“r(a  P^c^s  where  data  are  available  from  two  performance  classes  (e.g.,  flights  produced  by 
pi!°tSfand  fl  ghls  Produccd  »y  neophyte  student  pilots).  Candidate  proficiency  measures  will 

is  possible  to  test  these  sets  to  determine  if  they  come  from  different  parent  distribution*.  If  they  do 
come  from  different  parent  distributions  and  there  is  little  overlap  in  the  distribution  functions,  the^the 
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candidate  measure*  are  highly  iflcely  to  be  useful  In  proficiency  assessment;  i.e.,  there  is  a high  probability 
that  the  tested  measures  wfll  satisfy  other  validation  criteria.  On  the  other  hand,  if  these  two  distributions 
have  considerable  overlap,  then  the  measures  would  probably  not  be  very  useful. 

A test  for  determining  the  degree  of  similarity  of  the  two  parent  distributions  can  be  selected  from 
statistical  analysis  hypothesis  testing  where  one  can  initially  assert  that  the  two  distributions  are  the  same 
(null  hypothesis).  One  method  of  testing  the  hypothesis  that  distributions  F 2 (X)  and  F2  (X)  are  equal  is  the 
rank  sum  test.  This  is  a simple  non*pararaetric  test -which  indicates  the  likelihood  that  two  sets  of  data, 
which  may  be  of  different  sizes,  come  from  the  same  distribution. 

Teat  of  the  hypothesis  that  Ft(X)  is  equal  to  F2(X)  is  developed  as  follows: 

Let  Xj , Xj , . . . , Xn  j , and  Yj , Y2 , . . . , Y„a 

denote  random  samples  of  two  sizes,  nt  and  n2 , taken  from  populations  with  continuous  density  functions 
Fj(X)  and  F2(X),  respectively.  Let  these  two  sets  of  samples  be  ordered  in  increasing  magnitude  and 
combined  to  a single  ordered  set  where  a possible  arrangement  might  be  as  follows: 

YJfYJ,X1,Y3,X2,Y4,etc. 

Of  special  interest  is  the  sum  of  the  ranks  of  the  smaller  set  nt  (where  nj  < n2).  (For  example,  the  ranks  of 
the  X’*  are  3, 5,  etc.)  The  sum  of  these  ranks  is  a statistic  of  known  distribution  for  given  values  of  ns  and 
n2.  Therefore,  the  statistic  value  can  be  used  as  an  indicator  that  the  hypothesis  F,(X)  « F2(X)  is  valid. 
Table  13  gives  the  critical  values  or  limits  of  a 95  percent  confidence  interval  for  small  values  of  n2  and  n2 . 
(The  significance  level  of  the  rank  sum  test  is  not  preserved  if  the  two  populations  differ  in  dispersion  or 
shape.  Whether  or  not.  they  differ  in  this  way  is  expected  to  depend  on  the  measure  under  test.  Plans  to 
analyze  this  empirically  for  the  various  measures  on  the  T-37  problem  and,  as  required,  develop  techniques 
to  account  for  observed  effects  did  not  materialize  due  to  inability  to  collect  required  data.)  This  table  is 
taken  from  (Hoel,  1962)  and  applies  for  values  of  nx  and  n2  less  than  10.  For  larger  sample  sizes  the 
distribution  is  approximated  closely  by  the  normal  distribution  with  a mean  and  variance  given  as  follows: 

Meac^nt  (nt  +n2  + l)/2 

ns  n2  (nj  + n2  + 1) 

Variance  = 

12 


V.  SUMMARY  AND  CONCLUDING  REMARKS 

A computer-aided  system  has  been  developed  and  implemented  for  u*  in  deriving  and  validating 
measures  of  operator  performance.  Its  uniqueness  is  characterized  by:  (1)  a logical  division  of  human  and 
computer-processor  functions,  integrated  through  an  interactive  raan/machine  systems  approach  to 
measurement  research;  (2)  an  experimental  approach  to  deriving  measures  by  generating  vectors  which  span 
various  conceivable  measure  spaces  and  operating  on  the  vectors  using  multiple  regression  analysis;  and  (3)  a 
systematic  empirical  approach  to  validation-testing  of  candidate  measures  to  assess  their  likelihood  of 
contributing  to  overall  performance  measurement. 

One  of  the  most  important  features  desired  in  the  processor  is  its  ability  to  automatically  generate 
and  test  candidate,  performance  measures  with  a miniumum  of  inputs  from  the  user.  The  processor 
successfully  implements  this  desired  feature  in  that  it  reads  in  raw  performance  data  and  prints  out  tested 
performance  measures.  To  do  this,  it  first  automatically  performs  data  smoothing;  i.c.,  removal  of  noise  in 
the  data,  ft  then  performs  logical  sectoring  in  which  maneuvers  are  automatically  divided  into  sectors  that 
can  be  conveniently  analyzed  in  subsequent  processing.  Next,  the  processor  automatically  applies  a 
regression  analysis  procedure  to  establish  criterion  performance  in  the  form  of  simple  regression  functions. 
The  independent  variables  for  each  (Unction  were  selected  based  on  their  being  monotonic  over  the 
maneuver  sector  of  interest.  Finally,  the  processor  applies  adaptive  mathematical  models  to  the  data,  and 
bared  on  types  of  deviations  (BTS)  from  the  criterion  performance  functions,  generates  and  tests  for 
validity  a variety  of  performance  measures.  It  is  truly  an  automatic  processor. 
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Table  13.  Rank  Sum  Critical  Values0 


The  sample  sires  arc  shown  in  parentheses  {*»*,  ft?).  The  probability 
associated  with  a pair  of  etitical  values  is  the  probability  that  It  < smaller 
value,  or  equally,  it  is  the  probability  that  II  > larger  value.  These  prob- 
abilities are  the  closest  ones  to  .025  ami  .05  that  exist  for  integer  values  of  /?. 
The  approximate  .025  values  should  be  used  for  a two-sided  test  with  a « .05, 
and  the  approximate  .05  values  lor  a one-sided  test. 
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• Thii  tabic  was  extracted  from  a more  complete  table  (A*20)  in  /ntrwfuctio*  to 
Statistical Analysis,  2nd  edition,  by  \V.  J.  Dixon  and  F.  3.  Massey,  with  permission  from 
the  publishers,  the  McGraw-Hill  Book  Company. 
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i The  types  of  measures  -generated  and  tested,  were  defined  a priori  in  terms  of  underlying 

l characteristics  which  render  them  suitable  candidates.  One  type  (Relative)  represents  probability  measures 

l related  to  significant  event  proximities.  Another  (Absolute)  represents  measures  of  system  variable 

frequencies  and  periodicities.  A third  {State  transfer)  represents,  measures  of  state  transitions  that  occur 
over  time,  including  system  divergence  from  or  convergence  upon  criterion  terminal  performance.  A fourth 
\ type  (State  Frequency)  that  was  identified  but  not  specifically  addressed  in  this  effort  uses  data  generated 

! by  the  State  Transfer  computational  mechanism  to  address  measures  of  reference  function  deviations. 

To  investigate  the  above  measures,  vectors  were  identified  which  constitute  generators  of  the  measure; 
f spaces  corresponding  to  each  measure-type.  The  measures  thereby  spanned  are  explored  by  the  various 

computational  mechanisms  using  regression  analysis  and  a number  of  empirical  validation  tests.  Table  -14 
summarizes  the  measure  spaces,  the  components  of  vectors  by  which  they  are  spanned,  and  the  basic 
functions  performed  by  each  computational  mechanism. 


Table  14.  Summary  of  Measure  Spaces 
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Periodicities 
and  Response 
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Steady  State 
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Frequency* 
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Frequencies  and 
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Units* 
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Function 
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’Separate  computational  mechanism  not  yet  implemented  in  processor. 


The  success  of  the  automatic  maneuver  sectoring  is  a main  factor  in  processor  effectiveness.  It  allows 
use  of  simple  regression  functions  for  describing  criterion  performance  since  small  portions  of  the 
maneuvers  can  be  treated  separately.  Had  this  automatic  sectoring  not  been  feasible,  then  a considerably 
more  complicated  regression  function  would  have  been  required;  i.e.,  it  would  have  been  necessary  to 
attempt  to  model  the  entire  maneuver  or  large  portions  thereof  with  a single  regression  function. 
Preliminary  evaluation  of  the  automatic  sectoring  using  two  excellent  and  two  poor  maneuvers,  as  rated  and 
flown  by  IP’s,  indicates  that  the  sectoring  will  work  in  a satisfactory  way  over  a range  of  maneuver 
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demonstration!.  However,  since  the  beginning  of  each  sector  is  detected  when  a specific  variable  amplitude 
exceeds  a threshold  limit,  it  is  pdsslble  that  some  maneuver  demonstrations  (especially  those  produced  by 
neophyte  students)  presented  to  the  processor  will  not  be  processed  properly,  thus,  before  the  evaluation 
of  the  automatic  sectoring  can  be  considered  complete,'  distributions  of  these  key  variable  values  are 
required  over  the  range  of  expected  performance  demonstrations;  i.e.,  student  demonstrations  from 
neophyte  to  skilled. 

Another  key  area  in  the  development  of  the  processor  is  the  use  of  a simple  regression  function  that 
uaes  the  “five  sums  approach”  (Connelly  et  al,  1969, pp.  179-181),  so  that  additional  information  can  be 
simply  added' to  the  processor  as  it  becomes  available.  Initial  evaluation  using  two  excellent  and  two  poor 
demonstrations  of  each  maneuver  type  indicates  that  the  mechanism  for  developing  the  satisfactory 
criterion  function  is  available.  It  should  be  rioted,  however,  that  although  the  mechanisms  for  producing  the 
criterion  functions  exist  hi  the  processor,  the  data  itself  must  be  studied  using  additional  demonstrations  of 
flight  performance  in  order  to  determine  if  excellent  performance  data  is  clustered  about  the  criterion 
functions.  Such  clustering  is  necessary  for  establishing  useful  criterion  functions.  Should  clustering  of 
excellent  performance  data  fail  to  materialize,  as  evidenced  by  a large  residual  value,  maneuver  parameters, 
such  as  0m«x  and  4>Mix  may  have  to  be  included  in  the  regression  functions. 

The  adaptive  mathematical  models  developed  and  experimented  with  in  earlier  studies  have  now  been 
refined  and  adapted  to  use  in  an  automatic  processor.  Future  refinements  beyond  those  now  implemented 
may  be  easily  invoked  as  required  due  to  the  modular  design  of  the  software.  Whether  or  not  further 
refinements  are  necessary  or  desirable  could  not  be  determined  in  this  effort  due  to  the  previously 
mentioned  unavailability  of  sample  performance  data.  However,  the  central  features  of  what  is  believed  to 
be  powerful  and  highly  useful  measurement  research  tod  have  been  successfully  implemented;  and 
hopefully  the  underlying  theoretical  concepts  and  the  implementation  techniques  that  were  developed  and 
documented  herein  will,  as  a minimum,  serve  to  inspire  further  measurement  work  along  these  lines. 
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260.0 


FUNCTIONAL  FORM:  DV  = BQ  + B^  $ + + B^  + 

REFERENCE  FUNCTIONS 


Variable 


ir 


-3178.00 


18,06 


-0.05 


18.64 


-21S8.00 


-509.90 


-1.01 


0.05 


-0.57 


2.80 


73,60 


0.87 


0.00 


-0.89 


0,05 


.Jsmm£aS&£ 


MANEUVER  TYPE  Barrel  Roll 

ERRORS 


Dependent 

Variable 


#/samples 


SECTOR  6 


Excellent 

‘c 

101 

107 

21.48 

26.47 

16.48 

3.07 

3.32 

2.83 

i 

1.10 

1.51 

.70 

0.69 

.46 

.42 

.... 

Poor 


114  <L> 


100.83 

132.91 

210.67 

135.99 


~ 

11 

6 

-36.5  | 

-37.6 

-44.2 

349.0 

353.0 

348. C 

FUNCTIONAL  FORM:  DV  » B0  + B^+  8^2  + 

REFERENCE  FUNCTIONS 


Variable 


-$76.30 

-311.70 

-118.50 

-289.90 


-5.60 

-4.22 

0.58 

-4.36 


-0.01 

0.00 


B3 

B< 

14.63 

10.55 

21.86 

5.28 

29.96 

4.18 

18.90 

4.53 

i 

